Am I still working on the right thing?

The recurring question that has driven my research direction is whether I’m still looking in the right place. I’ve had two signals that have driven my suspicion that it may be time to pivot, and both are fairly simple. One suggests that an improvement I’ve made so far works reliably and well, but only on the exact setup I’ve tuned for it. That tells me I haven’t discovered the method that’s a step function up to solve a more generalizable class of problems. A far more serious signal is a failure mode: an undesirable outcome that persists no matter how I vary the things I can change within the current approach. In this case the risk is that the root cause of the problem lies outside the search space entirely.

Reflecting on my early research career so far, each transition in my own work has begun with one of those clues. I have kept searching until some failure revealed an assumption I hadn’t paid attention to before. At that point I changed course and studied it. The same pattern runs through each stage of my work: a progression from handcrafted rules, to learned models, to data and evaluation, and finally to making the thing that chooses experiments equally principled.

From handcrafted vision to learning

My first foray into research came as a fresh undergrad doing medical imaging, where I was using something that looked vaguely like traditional computer vision: heuristic image filters, elaborate pipelines of preprocessing steps, tuned by hand and adjusted manually whenever things stopped working. The problems felt achievable. Mostly they required enough effort to make all the pieces work together.

This changed as I got involved with SpecReFlow. Specular reflections are glare artifacts that can saturate regions of an endoscopy image and hide the tissue underneath, effectively blinding the camera in those places. I ended up using a U-Net to find them before reconstructing the hidden tissue, even though I had not really looked at how the model learned. I knew where the model belonged in the pipeline, but I couldn’t explain why it produced the behavior I wanted.

The handcrafted methods required a lot of problem-specific tuning, and each improvement stayed attached to one pipeline. Learning-based methods gave me a more reusable method to focus on, and the question shifted from assembling one solution to understanding how that solution could be learned.

When fusion exposed a data problem

At Modern Intelligence, the defense AI startup where I interned, a lot of my work focused on sensor fusion and multimodal learning. I was including multiple modalities in the data, but the models would often learn to use only one of them. Researchers in the field were adjusting gradients, changing the update rule, and generally trying to force every input to train. My own research process, though, became about identifying what the data actually said.

This led me to look more closely at theory of multimodal fusion and eventually working with MIMIC-III, a large public dataset of deidentified intensive-care records. One thing that struck me was how noisy, task-irrelevant, or unnecessary some of its inputs appeared. I didn’t publish that work, but I later noticed the same pattern in work by @pliang279 and his collaborators. MultiBench has a few surprises in its evaluation of the different methods on MIMIC-III. On the directly comparable tasks, simple late fusion is best or close to the strongest methods. GradBlend took about 56 times longer to train for a 0.3 point improvement in mortality accuracy over the reported runs. There was no gain on the disease-prediction metrics. A later analysis estimated near-zero multimodal synergy in MIMIC-III; most of the measured information sat in one modality.

In the case I was studying, the complicated optimization methods were trying to cure the symptoms of the problem. Gradient balancing can affect how learning proceeds from either relevant or irrelevant evidence, but it can’t force a reward for inputs that don’t matter. The root cause was the quality and task relevance of the data itself. None of this proves that the raw inputs lacked complementary information. That was when I started working on data research.

Accelerating the research loop

When I joined DatologyAI, I formed the vision-language model team and led it from zero to one while remaining an individual contributor. One of our first challenges was to make training and data loading sufficiently reliable that our ideas about how to curate the data might be tested at all. After we had a repeatable and reliable training unit, we moved on to improving the signal and efficiency of VLM evaluations. We built DatBench as a way of getting more decision signal out of fewer samples, so we could make enough curation decisions to build up a reasonable picture. We built pipelines to run those curation operations at scale. We had scaling ladders that let cheap runs of a model screen an idea before investing full compute. Our goal was to turn curation into a discoverable skill, one that could be learned systematically and reliably.

A TOML launcher gave us a shared interface (curate → train → eval) for researchers and agents; the launcher defined the universe of actions that could be taken and kept things comparable across runs. We made that launcher more and more flexible as we became more comfortable with the work it would need to support. DataSmith was a formal effort to build out this outer loop: propose intervention, run it, diagnose result, carry history into next decision. Under matched conditions in LLM post-training, the complete DataSmith system outperformed a general coding harness (Claude Code) in all 12 comparisons we ran; the average margin of success was 5.1 percentage points.

What does the future hold for me?

Some may call this laziness, but I’ve always wanted AI to take my job. I want to work at higher levels of abstraction. Once repeated work had a known structure, I wanted infrastructure to carry it (i.e. tweet below) so I could preserve attention for questions that still required human judgment. My goal is better decisions per experiment, not more experiments for the sake of it.

https://x.com/i/status/2072820055788679290

On the contrary, working at more abstract levels just amplifies whatever intent I give to AI. A faster harness can still optimize the wrong evaluation, and all sorts of new abstractions may be interesting, but they may not be useful. The question I keep returning to is whether I am still working on the real root problem, or simply continuing at the level I already know how to optimize.

The next few months will be critical to observe whether today's chosen abstraction of working on research will pay off. I'm betting it will and see an emerging market where owning your intelligence becomes commoditized. The only way to play this game at scale is to automate the research cycles themselves and bring humans to do the most critical work in order to make the highest impact decisions.