Process Book

Tengis Kelley & Tyler Dinh · CSE 4507 Final Project

Overview and motivation

Janelia, working with Google Research, recently released the complete wiring diagram, or connectome, of a male fruit fly's central nervous system. It contains every neuron in the brain and nerve cord and the synapses between them. Google Research used AI to turn flat electron-microscope images into 3D reconstructions. That sped up the mapping enough that larger brains, and one day a human brain, now look possible to map.

A connectome records which neurons connect and how strongly. However, a connectome does not show what happens in the fly's nervous system when it smells vinegar or feels wind on its antennae. Fly Sense lets a user explore that question. A user picks one of the fly's five senses and a stimulus, such as an odor, a sound or a touch. The site switches on the matching sensory neurons and runs a simulation of the signal spreading through the network. A 3D view further helps to show and inspect the signal moving step by step until it reaches the output neurons that move the fly's body.

We had two motivations. The first was curiosity. People online had already wired the fly connectome into games like Doom and Flappy Bird, and we wanted to use the fly's own senses and body instead. The second was practical. In the proposal we hoped that knowing which stimuli drive which outputs could suggest better fly traps, for example by smell or by sweet versus bitter taste. We also hoped that comparing fly and human senses could point researchers at neuron groups worth studying. The Questions section explains the questions we set out to answer and how they changed.

The site covers smell, taste, hearing, sight and touch, with 19 stimuli taken from published studies. Each sense is a circuit of 2,200 to 5,100 neurons, which was parsed from the roughly 166,700 annotated neurons of the full dataset.

Questions

The proposal asked how a fruit fly's output neurons respond when one of its five senses receives a stimulus. We planned to read the answer as the fly's predicted state or feelings. We also planned two questions. What kind of trap would best lure a fly, if we vary its smell or its sweet or bitter taste? Could fly senses suggest neuron groups to study in humans?

Changed questions

New questions

Data

Acquisition

We downloaded the full dataset, which are the three files provided by the Janelia male CNS v1.0 release (minimum synapse confidence 0.5). The file scripts/download.py fetches them into data/raw/, which is not committed to the GitHub repository because of its size.

FileSizeRowsWhat we use
body-annotations14 MB211,577 bodies × 36 columns superclass, class, subclass, type, entryNerve, somaSide, somaLocation
body-neurotransmitters43 MB1,835,518 bodies consensus_nt (falling back to predicted_nt)
connectome-weights1.05 GB151,856,684 connections body_pre, body_post, weight (synapse count)

Then, we keep the 166,700 elements that have a superclass annotation. That removes glia, orphan fragments and unannotated pieces. Between these neurons, 6,242,118 connections have at least 5 synapses.

Annotations

In the proposal, we expected to search the literature to find which neurons belong to each sense. The provided annotations in the dataset made this much easier.

Limitations

Stimuli

In the proposal, a CLIP model would turn a typed prompt into an input vector. For this milestone we use hand-picked presets instead: each one switches on a specific group of sensory neurons, chosen from the published literature. We checked every citation against the original papers and dropped the ones that did not hold up. For example, our first vinegar preset also included the DM4 and DP1m glomeruli. Neither is supported for vinegar, so the preset now uses only DM1 and VA2, as Semmelhack & Wang (2009) show.

SenseStimulusSensory neurons switched onBasis
SmellApple cider vinegarORN_DM1, ORN_VA2 (156)Semmelhack & Wang 2009
Geosmin (mold)ORN_DA2 (46)Stensmyr et al. 2012
CO₂ (fly stress odor)ORN_V (54)Suh et al. 2004
cVA (male pheromone)ORN_DA1 (184)Kurtovic et al. 2007
TasteMouth (labellum) touches foodlabellar bristle (147)Engert et al. 2022 (anatomy)
Legs step onto foodleg bristle (608)dataset subclass
Swallowing (pharynx)pharyngeal sensillum (48)dataset subclass
Taste pegstaste peg (58)dataset subclass
HearingCourtship song (low frequency)JO-B (87)Kamikouchi et al. 2009; Matsuo et al. 2014
High-frequency vibrationJO-A (50)Kamikouchi et al. 2009; Matsuo et al. 2014
Wind gust / gravityJO-C/E (475)Yorozu et al. 2009
SightBright flash / looming shadowR1–R6 (256)motion and luminance photoreceptors
UV lightR7 (377)R7 express UV opsins Rh3/Rh4
Blue lightR8p (87)R8p express blue opsin Rh5
Green lightR8y (107)R8y express green opsin Rh6
TouchFront leg touchedbristles entering by the prothoracic leg nerve (120)dataset entryNerve
Hind leg touchedbristles entering by the metathoracic leg nerve (432)dataset entryNerve
Back (notum) touchednotum bristles (200)dataset subclass
Whole body brushedall tactile bristles in the circuit (1,238)dataset class

Counts are the neurons a stimulus switches on in the built circuit. Large sensory groups are sampled (see Pruning and model decisions). Some counts are larger than a circuit's input layer: sensory neurons also synapse onto each other, so the pruning adds some of them at deeper layers, and a stimulus switches those on too. "Whole body brushed" turns on 1,238 tactile neurons; 623 are in the input layer and the rest sit in layers 1–6.

Data structures

The file scripts/build_subgraphs.py turns the raw files into one JSON file per sense in data/processed/, plus meta.json, which lists the senses, the pruning parameters, the simulation constants and the behavior labels. The website reads only these files. All five senses are built:

FileNeuronsConnectionsLayersInputsOutputsSize
smell.json2,54120,91576281641.5 MB
taste.json2,53024,75576042631.7 MB
hearing.json2,20320,22475892371.4 MB
sight.json5,12426,005116011262.3 MB
touch.json2,61118,87376232601.5 MB

Each sense file has the same structure:

{
  "sense": "smell", "label": "Smell (olfaction)",
  "sim":   { "steps": 9, "decay": 0.5, "gain": 4.0 },
  "nodes": [{ "id", "type", "superclass", "class", "side", "nt",
              "sign",       // +1 excitatory, -1 inhibitory, 0 modulatory
              "layer",      // hops from the sensory input (0 = input)
              "role",       // input | hidden | output
              "pos",        // normalized soma x/y/z
              "behaviour",       // output label (see Output labels)
              "behaviour_basis", // "literature" | "target"
              "region" }],       // body part, e.g. "front_leg"
  "edges": [{ "s", "t", "w",   // w = synapse count
              "wn",            // w / all synapses onto t
              "sign" }],
  "stimuli":   [{ "id", "label", "valence", "source", "neurons": [ids] }],
  "reference": { "<stimulus id>": { "top_outputs": [...] } }
}

reference stores the Python simulation's result for every stimulus. The website's JavaScript simulation is checked against it (all 19 stimuli match).

Pruning and model decisions

Our proposal noted that the whole CNS dataset would be too large to show or simulate in a browser, so we would fix this by cutting out one circuit per sense. This section covers how we implemented the pruning and how a stimulus can travel through the circuit. All settings are in the file scripts/config.py.

The circuit

  1. Inputs (layer 0). The sensory neurons of the sense, at most 600. Larger groups are sampled in proportion to each cell type, and the neurons used by a stimulus preset get priority, so small presets keep all their neurons.
  2. Grow outward one layer at a time, up to 6 layers (10 for sight). A downstream neuron is kept if at least 3% of all the synapses it receives come from neurons already in the circuit, and each layer keeps the 450 highest-scoring ones. Connections with fewer than 5 synapses are ignored.
  3. Reserve room for the outputs. Descending and motor neurons collect input from many places, so a sense rarely provides 3% of it. They get a separate budget (80 per layer) and a lower threshold (1%).
  4. One budget per stimulus. Each stimulus is grown separately with its share of the budget, and the results are merged. In our first version all stimuli shared one budget. The strongest stimulus then took every slot: geosmin and CO₂ were left with no path to any output.
  5. Thin the connections. Each neuron keeps its 12 strongest incoming connections, so the browser draws about 19,000–26,000 lines per sense instead of every connection.

Sight needed extra work. The optic lobe is so dense that, with 6 layers, the circuit never left it and reached zero outputs. With 10 layers it reached one. Visual projection neurons carry signals out of the optic lobe, so we gave them the same reserved budget as the outputs. The sight circuit now reaches 126 outputs over 11 layers.

The sight circuit in the layered layout: photoreceptors on the left, eleven columns of neurons, orange excitatory and blue inhibitory connections, with three outputs listed on the right.
Sight, "Bright flash" (layered layout). The 11 columns are the layers. Even with the extra layers, only three outputs receive any activity, and DNa07 receives most of it.

Connections

NeurotransmitterSignReason
Acetylcholine+1The main excitatory transmitter in flies
GABA, glutamate−1Inhibitory (glutamate acts mainly through inhibitory GluCl channels in flies)
Histamine+1Used by photoreceptors. Strictly it is inhibitory, but treating it as −1 would stop all visual signals at the first synapse in our model.
Dopamine, serotonin, octopamine0Modulators. They change how circuits respond rather than passing a signal, which our model cannot represent.

Sensory neurons always pass their signal on (+1). Each connection's strength is its synapse count divided by all synapses the receiving neuron gets anywhere in the CNS, including weak connections and unannotated fragments. A value of 0.2 in simpler terms is then "this input supplies 20% of all the synapses the receiving neuron gets".

Model

We use a simple rate model. It was inspired by the whole-brain model of Shiu et al. (2024, Nature), which simulated spiking neurons over the fly connectome to predict feeding behavior; ours tracks only an average activity per neuron. Every neuron has an activity between 0 and 1:

start:      r = 1 for stimulated neurons, 0 for all others
each step:  input_j = Σ over connections i→j of  sign_i × strength_ij × r_i
            r_j     = clamp(0.5 × r_j + 4 × input_j, 0, 1)
            stimulated neurons stay at 1

The simulation runs for at least 8 steps, and for 2 more steps than the circuit has layers, so the signal can reach the last layer (9 steps for most senses, 13 for sight). Every step is kept so the website can animate the signal moving through the network. The same model runs in Python (simulate() in scripts/build_subgraphs.py) and in the browser (js/simulate.js), and the two agree to within 0.0001 on all 19 stimuli.

Gain

The gain (4) sets how strongly activity is passed on. We tried several values on an earlier version of the circuits (10 steps each), and no single value suited every sense:

GainSmell, vinegar: strongest outputTaste, labellum: outputs above 0.1
30.0523
50.25166
80.49248

At gain 8, smell finally produces clear outputs, but taste is saturated, with hundreds of outputs lit up. We chose a moderate gain, and the website shows each output's activity relative to the strongest output of the current run instead of on a fixed 0–1 scale. With the current circuits, smell's strongest outputs reach only 0.003–0.13, while taste and wind reach 1.0.

The smell circuit in the layered layout after the cVA stimulus: the top outputs panel lists DNp32 at 100%, DNpe002 at 83% and DNge041 at 64%.
Smell, "cVA (male pheromone)". The outputs panel ranks descending neurons relative to the strongest one (DNp32 = 100%). In absolute terms, DNp32 reaches only 0.13.

Predictions

The strongest output for each stimulus, from reference in the data files:

SenseStimulus → strongest outputs (activity)
Smellvinegar → DNb05 (0.048), DNc01 (0.047) · geosmin → DNp29 (0.003) · CO₂ → DNp29 (0.053) · cVA → DNp32 (0.128)
Tastelabellum → DNg67 (1.0) · legs → DNg103 (0.83) · pharynx → MN10, a mouthpart motor neuron (0.96) · taste pegs → DNge098 (0.68)
Hearingsong → DNge111 (0.38) · vibration → DNg09_a (0.85) · wind → DNge091, DNge111 (1.0)
Sightflash → DNa07 (0.029) · UV, blue, green → DNpe011 (0.028–0.039)
Touchfront leg → leg motor neurons (Tr flexor 0.13) · hind leg → sternal rotator motor neuron (0.09) · back → DNg20 (0.28) · whole body → DNge122 (0.46)

Different stimuli within a sense mostly drive different outputs, which is the core idea of the project. However, sight is the exception, with UV, blue and green all peaking at the same neuron, DNpe011.

Output labels

Outputs get one of two kinds of label, and our data files record which kind in behaviour_basis.

1. Tested behaviors ("literature"). We gave these only to output neurons whose behavior has been shown experimentally, and checked each source:

Cell typeBehaviorSource
DNp01 (Giant Fiber)Escape takeoffvon Reyn et al. 2014
MDN (Moonwalker)Walk backwardBidaye et al. 2014
DNg100 (BDN2)Walk forwardPugliese et al. 2025
DNp09 (P9)Walk forward / approachBidaye et al. 2020 (as "P9"; that P9 is DNp09 is likely but not confirmed)
DNa01, DNa02Turn / steerRayshubskiy et al. 2025 (DNa02 well supported; DNa01's role is less certain)
DNg62, DNge078 (aDN1, aDN2)Groom antennaeHampel et al. 2015
MN9Extend proboscis (feed)Gordon & Scott 2009; Shiu et al. 2024

We observe that the Giant Fiber and the grooming neurons show up in hearing, MN9 and the grooming neurons in taste, MN9 in smell, and the grooming neurons in touch. However, none rank in the top 10 outputs for any stimulus, and sight contains none of them. With only these labels, the outputs panel showed bare cell type names.

2. Body-part targets ("target"). Every other output is labeled with a body part using the dataset's annotations. This doesn't describe a tested behavior, so the labels say "Targets…" or "Moves…" rather than naming an action.

OutputAnnotation usedExample labels
Descending neuronssubclass: the region of the nerve cord the neuron targets "Targets front-leg circuits", "Targets flight circuits (neck, wing, haltere)", "Targets wing + leg coordination (takeoff, landing)", "Targets several motor systems"
Nerve-cord motor neuronssubclass: the muscle group "Moves the front leg", "Moves the wing", "Moves the abdomen"
Brain motor neuronsexitNerve: the nerve the axon leaves through "Moves the mouthparts (pharyngeal nerve)", "Moves the antenna"

The descending and motor neuron codes are defined by Cheong et al. (2024, eLife), who also describe the upper tectulum as a single flight motor system and the lower tectulum as a hub for behaviors that need wings and legs together, such as takeoff. We found no paper defining the codes for brain motor neurons, so for those we used the exit nerve: the pharyngeal and labial nerves supply the mouthparts, the antennal nerve the antenna, and the cervical nerve the neck. That mapping is our inference from nerve anatomy. Each label also has a region key (for example front_leg, flight, mouthparts), so the website can light up the matching part of a fly body.

With both kinds, 1,038 of the 1,050 outputs across the five circuits have a label, and so do 172 of the 174 top-10 outputs that receive any activity. The labels also give a sanity check on the model. Touching the front leg most strongly drives motor neurons that move the front leg. Touching the hind leg drives hind-leg motor neurons. Swallowing drives a mouthpart motor neuron, and wind and a bright flash both drive descending neurons that target flight circuits.

The weakness is that the most common label for descending neurons, "Targets several motor systems", doesn't offer too much information. Linking these outputs to specific behaviors would be the main open problem for the next milestone.

The smell circuit in the anatomical layout: neurons cluster in the brain, with a few descending neurons trailing down toward the nerve cord.
Smell, "Apple cider vinegar" in the anatomical layout. Almost the whole circuit is in the brain. The few points far below it are a handful of motor neurons and one ascending neuron in the ventral nerve cord.

Exploratory data analysis

First, we looked at the dataset tables to learn what the annotations could give us. Then, we built the circuits and looked at them in early versions of the explorer.

What we looked atWhat we foundWhat it changed
Neuron counts by class and subclass Each sense maps to one annotation filter (see the table in Data) We did not need a literature search to find sensory neurons, so the time went into choosing stimuli.
Coverage of somaLocation 67% of neurons have a 3D position. Nearly all central neurons have one, but no sensory neurons do. The anatomical heat map from proposal Option 2 became possible. We place sensory neurons at their partners' mean position.
receptorType values for taste neurons No sugar or bitter receptor labels Taste stimuli use body location, and we dropped the sweet versus bitter trap question.
Output neurons with a known behavior Only 9 cell types, and none in any top 10 We added body-part labels, and the outputs list styles them apart from tested behaviors.
First sight circuit (6 layers) in the layered view The signal never left the optic lobe and reached zero outputs Sight now has 11 layers and a reserved budget for visual projection neurons.
Gain No single gain suits every sense. Smell stays faint while taste saturates. Activity colors use a log scale, and output bars are relative to the strongest output.
Anatomical layout of each circuit 85% of the touch circuit is in the nerve cord, and nearly all of the smell circuit is in the brain We kept the anatomical layout as a full view next to the layered one.
The touch circuit in the anatomical layout: neurons placed at their soma positions form the elongated shape of the brain and ventral nerve cord, with active neurons glowing yellow.
The touch circuit drawn at real soma positions (anatomical layout), after "Whole body brushed". The brain is at the top and the ventral nerve cord below it. We observe that 85% of the circuit's neurons (2,217 of 2,611) are in the nerve cord.

Design evolution

Proposal sketches

The proposal explored three directions and then combined them. Click a sketch to open it at full size.

Hand-drawn brainstorm page listing input ideas and four candidate views, with reference images of a force-directed graph, colored brain renderings, a Sankey diagram, a layered neural network fed a banana, and the fly nervous system.
Initial ideas. The input was a typed prompt that a CLIP model matched to a tagged image.
Sketch of a typed prompt going through a CLIP model to a fly swatter image and a radar chart, above a Sankey diagram from five senses through sensory and motor neurons to behaviors, with a time slider.
Option 1, Sankey diagram. Band width shows signal strength, a radar chart shows how much each sense contributes to the input, and a time slider animates the signal. It is readable, but it loses the brain's spatial structure and recurrent loops.
Sketch of a fly brain and nerve cord colored yellow, orange and red by activity, with a preset dropdown, an output action of flying away with 83% confidence, and an optional 3D fly body.
Option 2, anatomical heat map. Activity is colored over side and front views of the nervous system, and the input is a preset. It was the most intuitive option, but at the time we had no neuron positions.
Sketch of a layered neural network in 3D with orange and blue connections, a small force-directed graph, and a dense, unreadable force-directed graph of an unfiltered network.
Option 3, 3D layered network. People in the field already read networks drawn this way. It gets noisy if the circuit is not pruned well, and node size can imply importance.
Final proposal sketch combining a text prompt, CLIP and a radar chart, a 3D network filtered to olfaction with a fly swatter as input, and two anatomical heat maps compared side by side with a Save Scenario button.
Final proposal design. A prompt or dropdown fed CLIP, and a radar chart showed the input. A filtered 3D network highlighted active pathways, an anatomical heat map showed activity in the body, and the output was a readable action.

Proposal vs current

ProposalNowWhy
Typed prompt matched by CLIP to a tagged imageA list of 19 preset stimuli, grouped by sense Each preset switches on specific sensory neurons and cites a published source. A per-sense CLIP score would not be able to determine which neurons to switch on, and tagging images by hand is infeasible.
Radar chart of the input vectorDropped. A short description sits under the stimulus list. Each stimulus belongs to one sense, so there is no multi-sense vector to show.
3D layered network (Option 3)Kept as the Layered layout The x axis counts hops from the input, so position shows how far a signal has traveled.
Anatomical heat map (Option 2)Kept in 3D as the Anatomical layout Soma positions made it possible. Both layouts use the same neurons and colors, and switching animates each neuron to its new position, so you can follow it.
Sankey diagram (Option 1)Not built The 3D views and the step slider already show the flow over time.
Readable action or feeling with a confidenceA ranked list of output neurons with labels The data supports body-part targets, not emotions or confidence. See Questions.
"Save Scenario" comparisonNot built The URL records the sense, stimulus and layout, so two runs can be opened in separate tabs. A built-in side-by-side view is future work.
Hover tooltips and drag to rotateBuilt Tooltips show cell type, label, role, layer, neurotransmitter and activity.

Design decisions

Implementation

The site is static HTML, CSS and JavaScript modules, with no build step. The Python pipeline writes one JSON file per sense. The browser loads that file, runs the simulation in js/simulate.js and draws the result with three.js in js/network3d.js.

Interface

  1. Sense tabs. Keys 1 to 5 also switch senses. The old circuit fades out and the new one grows in layer by layer.
  2. Stimulus list. Each stimulus shows its valence, and the selected one shows how many sensory neurons it switches on. Choosing a stimulus runs the simulation and plays it.
  3. Playback. Play and pause (Space) advance one step per second, and the slider or arrow keys (Left, Right) help to move forward or backward to any step. The colors flow in the network between steps, to indicate more of a continuous motion to the user.
  4. 3D network. Drag to rotate, right-drag to pan and scroll to zoom. A switch changes between the Layered and Anatomical layouts (L), a toggle hides connections (C), and a button resets the camera (R). The legend shows the activity scale and the connection colors.
  5. Hover. Hovering a neuron shows its cell type, label, role, layer, neurotransmitter and activity.
  6. Top outputs. The list shows the 10 most active output neurons and ranks them continuously with a smooth reorder at every step. Hovering a row draws a ring around the same neuron in the 3D view, and clicking it selects the neuron and moves the camera to it.
  7. Neuron details. Selecting a neuron dims the rest of the circuit and replaces the outputs list with its details. These are an activity sparkline over the run and the neuron's four strongest inputs and targets. Clicking an input or target selects it, so you can trace a signal backward or forward.
  8. About. The About panel (A) explains the page and lists every control.

Evaluation

Future improvements

Generative AI use

From the course policy:

"…Generative AI may be used for the final project. When generative AI is permitted and contributes meaningfully to your work, disclose that use as directed. Regardless of whether AI assistance is permitted, you remain responsible for understanding, testing, evaluating, and being able to explain the work you submit.

Required WashU academic integrity statement
In all academic work, the ideas and contributions of others (including generative artificial intelligence) must be appropriately acknowledged and work that is presented as original must be, in fact, original…"

We used AI coding agents (Claude) during this project to spend less time on tedious coding, writing, and debugging and much more on designing the visualization and the user experience. We both have experience with agentic coding, so we worked with the agents extensively to design and plan features. We also tried a multi-agentic message board approach for the first time, which can be seen in a GitHub issue in our repository (issue #1) where our two agents posted their progress and questions for each other.

We reviewed, tested, and finalized all work, and we can explain it. As a sanity check, the browser simulation matches the Python simulation on all 19 stimuli.

Data / Processing (Tengis's agent)

Frontend (Tyler's agent)