Visualization Design Lab

Beyond the Demo: Building Visualization Research Software That Lasts

Fri, 17 Jul 2026 00:00:00 +0000

This is a blog post based on a keynote given at the VisGap Workshop at EuroVis on Monday, June 8, 2026.

Building visualization research software is not the same as building a prototype, and success does not always look like long-term software survival. Success comes in many nuanced forms. Sometimes the artifact matters most. Sometimes the idea matters more than the artifact. Sometimes the right outcome is a commercial product. And sometimes success means becoming part of the community infrastructure. In the following, I’ll dive into five software projects and reflect on what worked and what did not. For each of them, my ambition starting out was to build software that would be widely used by a community. By that measure, some of these projects failed, while others succeeded.

Failure: Caleydo – Genomics Data Visualization

As a budding PhD student I fell in love with designing and implementing cool visualization systems. The things you can do are exciting, and doing engineering work made me feel like a real computer scientist. At the time, I worked very closely with Marc Streit – sitting in the same office and hacking away on Caleydo, our biomolecular data visualization platform.

Caleydo visualized biomolecular data, ranging from gene expression to copy number data to pathways and how they are impacted by experimental data. We had some wild visualization ideas, including a 3D “bucket” visualization connecting related items with visual links.

As part of the project, we published several papers, including a paper that got a 3rd best paper award at EuroVis on visualizing patient stratifications – intended to help in characterizing cancer subtypes and a paper in Nature Methods.

Caleydo StratomeX visualizing the overlap of clustered and categorical genomic data.

While this was a great playground for experimenting with visualization design, we also invested a lot of effort in making sure our software was usable. For example, we built sophisticated data importers and wrote user-facing documentation.

This illustrates that we were serious about building software that could be used by others. But despite our best efforts, Caleydo never really took off. The software was used by a few labs, but it never became a widely adopted tool. So what happened?

Caleydo was discontinued in 2014. There were many factors that led to its demise: First, we had an aging tech stack. Caleydo was built in Java as a desktop application, yet users were coming to expect web-based tools without the hassles of installing and updating software.

Also, we never really broke into the target community (biomedical scientists). We were outsiders to the genomics community, and we did not have the right connections to get our software in front of the right people, didn’t go to the right conferences and didn’t publish consistently in the right journals.

We also never had sustained funding to support the software, and as the core developers aged out of the project, there was no one left to maintain it.

A larger takeaway though is that building and maintaining GUI tools is CRAZY HARD. GUI tools have so much interaction complexity, which makes them hard to build, hard to test, and hard to maintain, which is a significant resource problem especially in open-source and academic settings. Consequently, there are only a few successful open GUI-based tools, such as Cytoscape or Paraview.

Even though it failed as software, Caleydo was still hugely successful as a research project. It produced many publications, led to more than five completed PhD theses, and helped launch two faculty careers (mine included). It also helped me learn a lot about software engineering, and how to build and maintain software in a research context.

Success: datavisyn – From Research Prototype to Commercial Product

Next, I’d like to talk about easily the biggest software success story in my career so far: datavisyn. datavisyn is a company that develops data visualization solutions for pharmaceutical and biomedical applications. It was founded in 2016 by Marc Streit, Sam Gratzl, Nils Gehlenborg, and myself, and is now led by Dominic Girardi, Michael Pühringer, Marc Streit, and many other talented individuals.

The company grew out of academic visualization prototypes, and it was motivated by a simple realization: if we wanted to build and sustain serious visualization software for real users, we needed an organizational form that could support long-term engineering, documentation, hosting, and maintenance.

It started as a ranking visualization for comparing weighted lists – LineUp. The paper was very well received, and the system addressed a real problem: people needed a way to compare rankings interactively, not just read them.

The initial version of LineUp was implemented in Java, but a crucial step was the transition from Java to the web. LineUp also found an audience outside academia. In particular, Marc and his collaborators in the pharmaceutical industry saw enough value in it to expand it to target a biomedical use-case, leading to Ordino, which is now one of the key products datavisyn sells. That was an important signal: the tool was no longer just a research prototype, but something that could support practical decision-making in a domain where ranking and comparison matter.

At the same time, LineUp exposed a broader tension that we had already started to feel in Caleydo: academic software development and academic training do not always align well (we also wrote a paper about that). We could not simply assign maintenance and bug fixing to PhD students, because they also needed to do novel research. The more a project depends on sustained engineering, the more that tension becomes visible. That problem became even more pronounced once we moved from “I want to build this myself” to “we need students to help maintain this.”

To address that, we started datavisyn in 2016. The idea was to create a setting in which the software could actually be maintained, hosted, documented, and extended in a sustainable way. datavisyn built a suite of tools around LineUp, and the web-based version remained part of the product. Over time, the company also had to make pragmatic choices about the code base itself: some parts stayed open source, while the commercial products became proprietary. That separation turned out to be harder technically than organizationally, and in the end the real question was not whether the code was open or closed, but whether the overall system could be maintained and evolved.

datavisyn has become a very successful business, with customers that include 7 of the 10 largest pharmaceutical companies. It has also grown into a team of around 40 people, which is a scale that would have been hard to imagine from the original academic project. That scale also makes it possible to build and maintain GUI software well, because the resources are there: there is dedicated staff for UI and UX design, engineering, testing, documentation – all of which would be nearly impossible to sustain in a purely academic setting.

What LineUp and datavisyn show is that a research idea can succeed in several ways at once. The idea can have academic impact. The software can survive by moving to a more sustainable technical stack. And the project can grow into a commercial setting that gives the engineering the resources it needs. That does not happen automatically, and it is not the right path for every project. But in this case, it turned a visualization idea into something that lasted much longer than a paper prototype would have.

Success: UpSet – Visualizing Intersecting Sets

UpSet is a very different kind of success story. It started with a simple but important problem: how do you visualize intersections between more than three sets? At the time, existing approaches did not scale well to the kind of set relationships we kept running into in practice. The original paper, published in 2014, was very well received, and the idea clearly hit a nerve in the visualization community. You can read my reflections on UpSet blog post that I wrote when UpSet received a Test of Time Award at IEEE VIS.

What made UpSet take off was not just the novelty of the visualization, but the fact that we met users where they were. In particular, the R version of UpSet fit naturally into the publication and analysis workflows of the life sciences community. That mattered enormously, because it made the method immediately usable in a setting where researchers already worked. The result was that UpSet did not just get cited heavily; it became a widely recognized way of thinking about set intersections.

Over time, UpSet also showed something important about research software: the idea can outlive the implementation. UpSet has been reimplemented multiple times, and at this point the various software versions are probably less important than the concept itself. That is not a sign of failure. On the contrary, it is a sign that the idea solved a real problem well enough to spread beyond the original code base.

Interestingly, UpSet was also not a project that was funded. Only in 2022 did we receive a grant that allowed us to build UpSet 2: a React-based version that can be used as a library or through a web interface for uploading data. UpSet 2 also added accessibility goals, including support for blind users. That sequence is telling: sometimes the strongest software ideas succeed first, and the funding follows much later.

So UpSet is a success story: It solved a real need, it fit the workflow of its target users, and the idea proved durable enough to survive multiple software generations. In other words, the lasting contribution was not just the artifact, but the concept itself.

Failure: MultiNet – Good Engineering Was Not Enough

At this point, one might think I had finally figured out how to build successful visualization software. We had learned from earlier projects, and in 2019 we received a substantial infrastructure grant to build MultiNet, our platform for visualizing multivariate networks. The goal was ambitious: professionalize the software development process, build a real platform, and create something that could support both research and use.

This time, the engineering side was much stronger. We worked with Kitware, an experienced open-source science software company, and we also hired professional software engineers ourselves. MultiNet became a real, working system with adjacency-matrix and node-link views for multivariate networks, and users could upload their own data. On paper, this looked much closer to the kind of sustainable research software project we had been trying to build all along.

And yet, MultiNet still did not really take off as a widely used network visualization platform. The problem was not the implementation. The problem was that the scientific momentum was weaker than the engineering momentum. The science team was never quite as invested in publishing novel network-visualization contributions, and there was not a strong enough core of people who felt passion for the project itself. Students rotated through, collaborators had specialized needs, and the project never developed the kind of sustained excitement that keeps a research software platform alive.

That was an important lesson for me: even if you get the engineering right, that does not automatically make the project successful. Research software also needs evangelists, committed users, and a community that keeps pulling it forward. MultiNet had solid infrastructure, but not enough social and scientific energy to become the kind of widely adopted platform we hoped for.

Success: reVISit – Building for our community

Finally, there is reVISit, our open user-study platform. Together with co-PIs Lane Harrison and Andrew McNutt, and a very capable team, we built reVISit as a platform for running open, reproducible visualization studies. Unlike some of our earlier projects, this one is much more tightly connected to the community we ourselves are part of, which makes a big difference: we can find users, testers, collaborators, and adopters much more easily because we are already in the same conferences, workshops, and discussions.

reVISit is also deliberately designed to avoid some of the classic problems of visualization software. Instead of exposing a large and fragile GUI for every aspect of study design, it uses a DSL for writing experiments, which makes the system more manageable and easier to maintain.

That focus has paid off. reVISit has seen meaningful early adoption, some community contributions, and strong scientific output, including a best paper at IEEE VIS. Just as importantly, the project has kept both the engineering and the research side engaged: there are now spin-out paper projects that build on reVISit or add new capabilities to it, which helps keep the platform alive in a way that pure maintenance work often does not.

At the same time, reVISit is not without challenges. We have tried to avoid offering hosted services for legal and practical reasons, but a server-based offering would certainly make the system easier to use. Similarly, a GUI for experiment design would lower the barrier to entry for less technical users, which may be critical as we attempt to branch into fields outside of computer science.

Funding is also an ongoing concern: the current support from the National Science Foundation is generous, but it will not last forever, and we need to think carefully about what comes next. Still, reVISit feels like a genuine success story because it combines a useful tool, a real community, and a research agenda that continues to produce new work.

Reflection

Looking across these projects, a few patterns become hard to ignore. Prototypes are not the same as visualization tools, and GUI software in particular is expensive to build, hard to test, and hard to keep alive. It ages quickly, and there are surprisingly few examples of large, successful open GUI-based tools. In practice, libraries are often the more realistic form factor: they are easier to reuse, easier to maintain, and can stay relevant even as the surrounding software changes.

The other major issue is incentives. Students need papers and short-term wins, collaborators want analyses and concrete results, and faculty have longer horizons but often cannot do all the implementation work themselves. That means the people doing the science, the engineering, and the maintenance are often not optimally aligned. When those incentives are misaligned, software projects become fragile very quickly.

That leaves sustainability as the central challenge. Continuous funding is difficult, and maintenance is rarely rewarded in the same way as novelty. Commercialization can solve part of that problem, but it is not a universal answer. In some cases, the best outcome is not a durable application at all, but a durable idea, or a library, or a community infrastructure layer that others can build on.

In the end, I do not think of these projects as a clean success or failure story. The outcomes have been mixed, the funding has often been precarious, and sustaining software over many years is genuinely hard. But I still find this work deeply rewarding. It is a rare privilege to build things with students, to work alongside professional engineers, and to move between academia and industry. That mix of perspectives is intellectually stimulating and, at its best, a lot of fun. It is difficult work, but it is worth doing.

ReVISit 2 Paper Wins Best Paper Award at IEEE VIS

Mon, 10 Nov 2025 00:00:00 +0000

Running online visualization studies is now standard practice in VIS and HCI research. Yet the process remains fragmented: researchers stitch together survey tools, custom web code, logging scripts, analysis pipelines, and ad hoc debugging workflows. Users of reVISit already know this story: reVISit consolidates this ecosystem into a single open framework that supports the entire experiment life cycle – from design to dissemination.

To inform the academic community about the new developments in reVISit 2 – which we already described in these blog posts we wrote an academic paper about it.

Positioning reVISit in the Ecosystem

In the paper, we first situate reVISit among existing study platforms. We compare it to commercial survey systems, domain-specific research tools, and library-based frameworks. While survey platforms excel at rapid deployment, they rarely support sophisticated interaction logging or fine-grained experimental control. Academic tools often address specific domains or slices of the workflow, but lack long-term maintainability or broad adoption.

ReVISit 2 is designed differently: it treats experiment design as programmable infrastructure. A JSON-based domain-specific language (DSL) models sequences, blocks, counterbalancing strategies, interruptions, skip logic, and dynamic control flow. On top of that, reVISitPy provides Python bindings that allow researchers to generate complex study configurations directly from notebooks. The result is a framework that emphasizes expressiveness, reproducibility, and ownership over one’s experimental stack.

We also describe technical advances in reVISit 2, including first-class Vega support, automated provenance tracking, participant replay, and improved debugging tools such as the study browser. These features aim to tighten feedback loops during piloting while preserving transparency during dissemination.

Putting it to the Test: Replication Studies

To demonstrate that these capabilities are not merely architectural, we conducted a series of replication studies.

Each study highlights a different capability of the system. In one, we implement adaptive and staircase-style designs to evaluate visualizations of correlations using dynamic sequencing logic, showing how complex control flow can be expressed directly in the study configuration. In another, we integrate think-aloud protocols by embedding audio recording and transcription into browser-based experiments, allowing researchers to capture reasoning during interaction rather than only after the fact. Finally, we demonstrate provenance tracking and replay by instrumenting interactive visualizations to capture detailed interaction histories, enabling fine-grained participant replay and qualitative analysis. reVISit 2 also provides deep linking to specific trials or moments in user studies, to aid in dissemination and transparency. For example, this link takes you to the exact state you see in the above image.

Across these replications, we recruited hundreds of participants and reproduced key findings from prior visualization studies. Just as importantly, the studies surfaced practical lessons about counterbalancing, recruitment logistics, and ongoing challenges in deploying sophisticated designs online. The replication work serves as both validation and stress test for the framework, and as real live examples for studies (including the data) for the community to learn from.

Across these replications, we recruited hundreds of participants and reproduced key findings from prior visualization studies. Just as importantly, the studies surfaced practical lessons about counterbalancing, recruitment logistics, and the realities of deploying sophisticated designs online. Together, they function both as validation and as a stress test of the framework. They also remain publicly accessible – complete with study configurations and data – serving as concrete, real-world examples that the community can inspect, reuse, and learn from.

What Users Told Us

We also interviewed experienced reVISit users to better understand how the system performs in practice. Several themes emerged:

Tighter development loops. Users appreciated the integrated development environment and the study browser, which makes it possible to jump directly to specific trials without stepping through an entire study.
Expressiveness over convenience. While the DSL requires programming knowledge, users valued the flexibility it affords – especially for mixed designs and adaptive sequencing.
Learning curve trade-offs. ReVISit inherits complexity from modern web tooling (e.g., React, TypeScript). This can be a barrier for less technical researchers, but it also enables deeper customization and extensibility.
Open infrastructure. The ability to fork studies, inspect core code, and maintain version stability was frequently cited as a strength, particularly for reproducibility.

Overall, feedback confirmed that reVISit is most effective for technically oriented research teams who need more than a survey builder.

Recognition at IEEE VIS

We were very honored to receive an IEEE VIS Best Paper Award for this work. Zach Cutler presented the paper on the main stage in Vienna.

This recognition reflects years of iterative development, community feedback, tutorials, documentation work, and, most importantly, the researchers who have trusted ReVISit in their own studies.

ReVISit 2 is not the endpoint. It is infrastructure for a research community that increasingly relies on sophisticated, reproducible, browser-based experiments. We look forward to continuing to build it together.

Learning with Hands: Making Complex Chart Types Accessible

Fri, 08 Aug 2025 06:00:00 +0000

Tactile Charts as Templates for Chart Learning

Tactile charts—charts you can explore with your hands by touch—are not new. They have long been used to represent data, helping BLV readers grasp spatial patterns and relationships. But they have limitations: they take time and resources to produce, can’t easily be updated, and are often harder to access compared to digital alternatives such as alt text.

So why do we use them? Because when it comes to learning a chart type—building that mental model that helps future interpretations—touch can be a powerful sense. And unlike data-specific tactile charts, template charts don’t require frequent updates as they are learning tools, not meant to communicate a specific dataset.

Working closely with our blind collaborators and following an iterative design process, we created educational tactile template charts for four complex chart types frequently found in scientific publications: UpSet plots, clustered heatmaps, violin plots, and faceted line charts (inspired by genome browsers). Each design is accompanied by digital exploration instructions to support independent learning. You can see our final tactile model designs and accompanying exploration instructions on our accessible website.

Do Tactile Charts Help?

We tested our designs of violin plot and clustered heatmap in an interview study with 12 BLV participants. Each participant learned two advanced chart types—one using our Tactile+Text method (a tactile chart with a textual description) and one with Text-Only. Afterwards, we gave them new datasets described only with alt text to see if they could apply what they learned to new datasets differently between the conditions.

The results were striking. Participants not only preferred the hands-on learning experience of tactile charts, but also reported that tactile charts helped them build a much clearer mental model of the chart types they learned. Tactile charts allow them to “picture” layouts and chart element shapes more vividly. Several described the experience as like being able to see the chart. Participants also use tactile charts as a reference in their minds to interpret new alt text descriptions more confidently.

Why Does This Matter?

In our interviews, participants spoke openly about the barriers BLV individuals face in visualization education and communication, and their desire for equal access to information. They saw tactile template charts as a way to bridge that gap, offering a more effective and inclusive approach to visualization education.

In a world increasingly driven by data, data literacy means a lot, such as job opportunities or the ability for equal information acquisition. By using tactile template charts as educational tools, we can equip BLV learners with transferable knowledge—knowledge that carries over to new contexts. Our hope is to empower BLV individuals to be better able to interpret unfamiliar visualizations and to participate fully in discussions with sighted collaborators in academic, professional, and civic discussions where data plays a central role.

We make our models available on the project website, so that anyone with access to a 3D printer can make their own template charts.

Sometimes, the best way to learn a chart is to feel it.

This work was generously supported by the Chan Zuckerberg Initiative.

VDL is Moving to Austria

Wed, 30 Jul 2025 06:00:00 +0000

After an incredible decade at the University of Utah, the Visualization Design Lab (VDL) is entering a new chapter: the lab is partially relocating to Graz University of Technology (TU Graz) in Austria. This will be a gradual transition: only Alex will be moving, while the rest of the team remains in Utah. For the foreseeable future, the lab will operate across two continents.

VDL’s new home in Austria will be the Institute of Human-Centred Computing at TU Graz.

Alex will be hiring a PhD student and a Postdoctoral Fellow in Graz. Check out the positions page for details.

A Brief History of the Visualization Design Lab

The Visualization Design Lab was founded when Alex joined the University of Utah and teamed up with Miriah in 2015. From then until 2021, Miriah and Alex co-directed the lab, leading numerous projects and mentoring many students.

In 2021, Miriah moved to Linköping University in Sweden, where she founded the Visualization and Interaction Design Group.

In 2025, Alex returned to his alma mater in Graz to continue VDL’s research mission at TU Graz.

This website will reflect the new phase of the lab in Austria, while continuing to preserve the history, people, and publications from the Utah chapter. For archival purposes, the final version of the Utah VDL site will remain available at
https://vdl.sci.utah.edu.

Making Data Visualizations Talk: Accessible Text Descriptions for UpSet Plots

Sat, 07 Jun 2025 06:00:00 +0000

Scientists love data. In fact, we love data so much that when you open up any journal article, chances are many of the pages are scattered with data visualizations. That is not just because we like pretty pictures (although we do!). Data visualizations are a powerful tool to show trends or patterns in the numbers that may be difficult to explain in words alone.

But what if you can’t see the data visualization?

Most charts are entirely inaccessible to people who are blind or have low vision. These users often rely on screen readers to read the contents on a computer screen aloud. However, those tools usually come up short regarding data visualizations. Instead of describing the chart, they might just say something like “Image of Figure 1.” That leaves the figure caption as the only source of information; yet captions almost never explain what the data is actually showing. Imagine trying to understand a chart with a title and maybe a vague summary. That is the reality for many blind readers.

One chart type where this really matters are UpSet plots, shown in the figure above. It is a popular way to visualize how different sets overlap. Think of it as a more organized way to show set overlaps than a traditional Venn diagram. These plots are used across disciplines, especially in computer science, bioinformatics, computational biology… practically any data-heavy field. But UpSet plots are visually complex and nearly impossible to interpret without sight.

That is where our work comes in. We developed a system that automatically generates screen- reader-accessible text descriptions for UpSet plots. But not just generic summaries like “there are bars and dots.” These descriptions highlight key structural features of the plot, including the largest intersections, distribution of elements, and other patterns typically interpreted visually. The goal is to give screen-reader users meaningful access to the same insights a sighted reader would glean from the visualization. It basically provides the plot with a narrative voice.

What makes our system unique? This is not just a one-off tool for a specific dataset. We analyzed 80 UpSet plots from published papers and identified recurring data patterns, such as skewed intersection sizes or dominant sets that shaped the narrative focus of our text descriptions. These insights informed a modular, rule-based pipeline that produces both brief summaries and extended descriptions. As shown in the annotated example below, the generated text incorporates multiple layers of information, from basic chart elements to statistical summaries.

But do these text descriptions actually work?

To find out, we conducted in-depth interviews with 11 blind or low-vision screen reader users. And these were not quick usability tests! They were in-depth conversations where participants walked through the descriptions, shared what made sense and what did not, and suggested changes that made the description stronger.

These interviews revealed valuable insights into how experienced screen-reader users navigate complex text. For example, our participants emphasized the importance of structure, recommending using bullet points to help navigate longer descriptions. They also highlighted the need for high-level takeaways before discussing fine-grained details.

After this process, Maggie McCracken, one of the lead authors of our study, realized how powerful participation itself could be. As a psychology researcher, she has worked with many participant groups, but this one stood out for how genuinely enthusiastic they were. They were not just excited about the tool but about the fact that this type of research was happening at all. Several told us they were entirely used to being left out of data visualizations and that being part of the process was meaningful. And that says a lot, considering the data they were reading was about COVID symptoms, which is not exactly edge-of-your-seat material.

So yes, making charts more accessible helps blind readers. But more than that, it improves how we all think about communicating data.

If you are curious, you can try it out for yourself at https://upset.multinet.app. You can also upload your own data and generate descriptions instantly.

This project serves as a reminder that accessibility is not just a technical challenge. It is about asking who gets to be included in scientific conversations. Accessibility is not just a bonus feature but part of that conversation. Charts included.

This work was generously supported by the Chan Zuckerberg Initiative.

Max Lisnic Successfully Defends Dissertation

Thu, 15 May 2025 11:00:00 +0000

Max Lisnic successfully defended his dissertation on “Designing Resilient Visualizations Toward More Accurate Data Discourse”. Max was co-advised by Marina Kogan and Alex Lex. The committee was completed by Kate Isaacs, Vineet Pandey, and Crystal Lee.

Max will join WPI as an assistant professor this fall!

Congrats, and good luck with your next steps!

Reading Between the Lines: The US Computer Science Graduate Admission Process

Wed, 09 Apr 2025 06:00:00 +0000

The time between submitting your application and hearing back can be stressful, in particular if you don’t hear back for a long time. This post is meant to help you read between the line, so that you can judge what a non-response from a grad program means. I’ll talk mostly about PhD admissions, but sprinkle in some tidbits for MS admissions. I’ve previously written about what you can do to get into grad school.

It’s March or April. I haven’t heard back at all. Is there still a chance that I will get accepted?

To be blunt: it’s unlikely. US PhD programs often have three piles: definite admits, students on the waitlist, and rejects.

Unfortunately, only the definite admits will hear about a decision early, everybody else is kept in purgatory for a while. Our CS program informs definitely admitted candidates as early as late January or the first half of February. Domestic students are invited to visit campus in mid-February; international students are invited to participate in a virtual event at the same time.

So, if you haven’t heard back from a PhD program by the end of February, it means that you’re not in the “definitely admit” pile. There might still be hope though: you might be on the waitlist.

For MS students, the timeline is usually a bit longer: you should expect to hear in February or March.

Candidates on the waitlist are meant to fill slots that free up as students in the “definite admit” pile reject an offer of admission. Waitlisted candidates are considered promising students, but ranked slightly below others. Not every school has a waitlist; Utah only introduced one a few years ago. You might be told that you’re on a waitlist, or you might not hear back at all. Sometimes, waitlisted candidates may even hear back the latest: definite accepts go out first, rejects go out next, but wait-listed candidates may be in purgatory for a while. Whether a slot opens up when you’re on the waitlist is difficult to predict – it typically depends on whether other candidates who are vying for a slot in a particular area / with a particular professor accept or reject an offer.

Most schools don’t have a waitlist for MS students.

Can I e-mail admissions to learn more about whether I’ve been accepted?

Typically, an e-mail to a generic admissions contact or the director of graduate studies will not help you learn about your status. We usually don’t respond to these inquiries because we wait for official channels to give you the news instead.

Can I e-mail the professor I want to work with to learn whether I’ve been accepted?

You have better chances (though still slim ones) to get a response about your status from a professor you’ve interacted with, or who you listed in your letter as a potential advisor. Generally, individual faculty have a lot of influence over the admissions process. If they want to work with you (and have the money / time), you’re almost certainly going to be admitted.

This doesn’t hold for MS students: since most MS students typically focus and coursework and don’t interact closely with individual faculty members, it’s unlikely that a particular professor will know anything about your status, and professors (outside of the admissions committee) also don’t intervene on MS admissions decisions.

I’ve been rejected by a program. Can I ask for feedback on how to improve my application?

You typically won’t get individualized responses to an inquiry about what you can do to improve your application. That’s partially because it would take a lot of effort to respond (Utah’s CS program gets more than 1000 grad applications), and partially because people worry that you might use any advice in litigation against them. If you’d like feedback on your application, you should reach out to your letter writers who know you personally.

I’ve been accepted by a program, but haven’t heard back from my ‘dream school’ yet. When do I have to make a decision?

All major grad schools have agreed to use the same deadline by which students must accept an offer of financial support (that usually comes with a PhD offer): April 15. It’s strongly advisable to accept any offer you may have by that deadline. You might still be admitted if you respond later, but there are no guarantees anymore. Typically an offer letter lists a specific deadline.

We (at Utah) try to give offers to students on waitlist sooner, as we learn from rejections by other students.

I’ve accepted an offer but now I received a better one. Can I walk my acceptance back?

It’s not great but it happens. No one can force you to attend a PhD program, so yes, go ahead and rescind your acceptance, but do it as soon as possible.

I’ve been accepted to multiple programs. Should I decline offers I won’t take immediately?

Yes! The sooner the better. You might give a waitlisted student a chance to get admitted.

Professors keep ignoring my e-mail. What can I do?

I’m sorry, it sucks to be ghosted. But most professors delete a lot of e-mail from prospective students. Here’s how to get a response:

Write only targeted e-mail to people you want to really work with. Don’t mass e-mail, we can spot mass e-mails (even if you paste a paper title of ours).
Keep it short and to the point. Five sentences at most.
Don’t chat-gpt us. We don’t care for a wall of very well written but empty text.
Don’t get our research area wrong. This will almost certainly get your e-mail deleted.

Reflections on UpSet

Wed, 16 Oct 2024 10:00:00 +0000

UpSet has won the 10-year Test of Time Award at IEEE VIS. I’m deeply honored by this award, and I want to thank the committee for choosing our paper. But most importantly, I want to thank my co-authors Nils Gehlenborg, Hendrik Strobelt, Romain Vuillemot, and my PostDoc advisor Hanspeter Pfister!

As of October 2024 the UpSet paper has been cited 1900 times. Together with Jake Conway and Nils Gehlenborg, I also wrote a short follow-up paper introducing an R version of UpSet, which has been cited 2500 times. So clearly, those papers have hit a nerve – UpSet is now the de-facto standard for visualizing set data with more than 3 sets. Also, UpSet has been re-implemented many times, making it available on various platforms and in many programming languages, which certainly contributed significantly to its success.

In this post, I want to first give a brief history on how UpSet came to be and acknowledge the giants on whose shoulders we stood, and then share my thoughts on what made UpSet successful.

How UpSet Came to Be

The intention of UpSet plots was to solve the problem of set visualizations for more than three sets. Very specifically, it was inspired by the now infamous six-set banana
Venn diagram.

This chart was widely ridiculed on Twitter back then. When I saw this, I figured that we should be able to create a better visualization for set data than that. So I did some research and found various set vis methods, including radial sets. But the method that I liked the most was a response to a “Vennerable Challenge” for Autism data by Robert Kosara:

I liked that this chart uses a simple bar chart instead of the irregular shapes that Venn diagrams use, but I struggled a bit with parsing the tree – it always had to trace paths to the root and I couldn’t spot any trends with regards to the set intersection patterns.

Also, this chart doesn’t actually plot intersection sizes as bars but a different attribute (% of correct diagnostic tests), so it’s actually visualizing something very different.

Still, this chart got me thinking, and shortly before Christmas 2013, while I was a PostDoc in Hanspeter Pfister’s group at Harvard, I sketched this in my notebook:

As you can see, all of the principal ideas of UpSet are already in that sketch: the matrix layout for sets, the bar charts for set and intersection sizes, even advanced metrics such as “deviations from expected intersection sizes” and grouping and aggregation.

I pitched the idea to my office-mate at Harvard, Nicolas Bonneel (who is a computer graphics researcher), and he immediately got the idea, but was not very excited because it “was so simple”.

I went home for Christmas and kept thinking about it and decided to give it a go for VIS. I also wanted to do this on the web; I had only done Java-based visualizations up to this point, but there was this hot new thing called D3 that I wanted to take out for a spin. So I started out and pretty quickly got to a first prototype that you can see here:

At this point I realized that I’ll need help to pull this off, especially with only three months to go till the conference submission deadline. My advisor, Hanspeter Pfister, was supportive, so we assembled the “dream team” consisting of my PostDocs friends that were also doing visualization research at Harvard: Nils Gehlenborg (at the Harvard Medical School), Hendrik Strobelt (who had just joined the lab that month), and Romain Vuillemot (at the Harvard Kennedy School).

Everyone on this team was an experienced visualization researcher, and magically, everyone found the time to really pour their heart into the project. We had many lively discussions on our wall-spanning whiteboard, and lots of heated arguments about UpSet features in its early days.
We were able to pull the UpSet paper off in just three months – from idea, to design and refinement, to evaluation, to writeup – and you can still admire the result here: https://vcg.github.io/upset/

As you can see this is actually a fairly complicated interactive visualization system. While the basic design remained the same, everyone contributed immensely to make it all come together. It took a million design decisions to actually make this work. We also used interaction extensively to enable users to answer a variety of questions. The system includes supplementary visualizations, queries, aggregations of intersections, attribute visualizations, and so on.

We made the deadline and properly celebrated the submission!

The paper received mostly strong reviews on its first submission to InfoVis.
Of course, the dreaded “Reviewer 2” found that “The proposed software and its associated methodology does not go beyond what this area has been providing as standard for the last 15 years.” Nevertheless, the other reviews were sufficiently positive, and we were happy to present it that year in Paris at the 2014 Visualization conference.

What Made UpSet Successful?

So, what made UpSet more successful than some of my other visualization papers? I think it was a multitude of factors. Some of them might be unique to this project, but others may be relevant as a lesson also for other visualization projects.

UpSet Solved a Real and Pressing Need

There really wasn’t a good way to visualize set intersections of four, five or even more sets. Existing Venn or Euler diagram solutions just don’t work, it gets too complicated to understand what sets are involved in an intersection, and area-proportionality is incredibly hard for Venn diagrams with more than three sets. And especially in the biomedical domain we saw a lot of bizarre many-set Venn diagrams:

Other solutions required interaction, or were difficult to interpret. In contrast, UpSet plots are at least somewhat self-explanatory and solve the problem.

UpSet Met Users Where They Are

If we had just stopped with our UpSet implementation where we were with our InfoVis paper, I doubt that it would have been a big success. I’m sure people in our community would have appreciated it, and might even have built extensions based on our open source code. But we realized that for broad adoption, we have to meet users where they are, and we have to communicate with them.

So we set out to build the R version of UpSet – UpSetR. The idea was that the biomedical community is already working with R for their data visualization needs. Uploading data to our web tool and then taking a screenshot would have yanked them from their workflow, and would have resulted in a “dead-end” plot that they’d have to manually adapt if their data changed. An R version would make it simple to just integrate UpSet plots in their current workflow, significantly lowering the barrier to its use.

We also promoted the UpSet, and particularly the R version in two ways: we wrote a short two-page “Applications Note” in a widely read bioinformatics journal. There was no new scientific content (as judged by the VIS community) in that article, it was just a simplified re-implementation of the tool in R.

And then we also wrote a how-to guide style article for a then popular series: Nature Method’s “Points of View”. In that article we talk about set visualization in general: for example, we recommend using Venn diagrams for two or three sets, and matrices for very large numbers of sets. But we also point to UpSet plots for the “middle ground” of 4–10 sets.

At a later point, we also set up UpSet.app to document the general use of UpSet and published a Wikipedia article on UpSet. It was a small victory when the Wikipedia article was approved for inclusion by the Wikipedia admins instead of being deleted for “lack of notability”.

A Simple Idea is More Impactful than a Complicated System

While our IEEE InfoVis submission was a complicated system with sophisticated interaction. Our team, and our reviewers, are visualization researchers, and we like to think through all of the things that we can do, and how we can leverage interactions and have cool animations.

However, the version of UpSet that really took off is not our web-based version, but the more basic one that is implemented in UpSetR:

While some features, such as sorting made it into UpSetR, others didn’t. That basic idea is simple enough, and people started using and citing it in their papers. Soon we saw other versions of UpSet pop up, for example in Python, or even a different (better) R version. At this point we know of 13 re-implementations of UpSet – so clearly, the idea transcended the implementation.

While I do think software tools and libraries add tremendous value to our community, and my team invests a lot of energy in building and maintaining software, it is certainly nice to see that an idea takes off, not at least because you don’t have to maintain the software yourself!

UpSet is Useful for Communication, Not Only Discovery

Finally, UpSet hits the sweet-spot between being useful for discovery / exploration, but also for communication. Like scatter plots or histograms, UpSet can be used in the discovery process, but is still simple enough so that they can be used as a figure in scientific articles.

Nils often complained to me that he believes that a lot of complicated visualizations don’t get “credit” for the parts they play in the discovery process, but rather give ideas that are later tested using statistical methods or simpler plots; while the visualization tool that lead to the insight doesn’t get credit (for example, in the form of a citation).

This isn’t the case of UpSet plots, which are easy for scientists to generate, given the many implementations. And unlike our web-based tools, the static Python and R tools generate “paper-ready” figures, without UI elements that need to be edited out or other issues that make it difficult to include a figure. It probably also helped that we clearly instructed users on how to properly cite UpSet.

The Future of UpSet

While I’m certain that the reason for UpSet’s success is its simplicity, I still haven’t abandoned the interactive UpSet visualization system. In fact, we’ve just released UpSet 2.0 – a version that has the same analytical features as the original UpSet, but supports data upload, public sharing of interactive UpSet plots, and can be integrated in other tools as a React component. It also has a full provenance history (including undo/redo). And it’s enabling new research: for example, we are now generating (and evaluating) text descriptions for UpSet plots, so that they can become accessible to blind and low-vision users.

In closing: thanks again to my co-authors and friends for helping bring this project to life, and to the community for this recognition.

Lessons Learned from Visualizing Multimodal Data... with Aardvarks...

Mon, 30 Sep 2024 14:00:00 +0000

Understanding cancer is the key to fighting it.

Cancer is a terrible disease caused by your cells growing out of control. If we can understand exactly how cancer cells grow, move, and divide, we can develop strategies to prevent, diagnose, and treat cancer.

A fundamental way to further our understanding is to collect data to represent how cancer cells are growing. It turns out that this is pretty hard because…

Cancer is complex — we need complex ✨ multimodal ✨ data to represent it.

What is ✨ multimodal ✨ data? For us, it is data in different formats that represent different aspects of the same phenomenon. Specifically, we work with 🌌 images, 🌳 trees, and 📈 time-series data.

How is this data collected?

The datasets we worked with are from live-cell microscopy imaging; in other words, 🌌 images of cancer cells are recorded over time as those cells grow and divide.

Algorithms can track individual cells over time based on their position and other characteristics, such as size.

Then, derived attributes based on these images can be computed, such as the cell size or the amount of a specific protein in a cell. These attributes are calculated over time, resulting in 📈 time-series data.

During these experiments cells might divide into two daughter cells. If we record these divisions, we can construct a 🌳 tree of cell relationships or cell lineage.

Now that we have the data, we need to try to understand it. It turns out that…

Understanding multimodal data requires us to think about all the modalities together… and this is hard.

Each of the data modalities (🌳🌌📈) is necessary because they capture a different aspect of how cancer cells develop:

The 🌌 images show the spatial relationships between cells.
The 📈 time-series data shows how cells grow and change over time.
The 🌳 tree captures how cell attributes propagate across generations.

But one thing that quickly became clear was that to fully understand the phenomenon of interest (the spread of cancer cells), we needed to synthesize all three of these modalities together.

Right now, researchers are synthesizing or combining data modalities manually. In other words, they look at an image, then at time-series data, then back at an image, then at a tree, and mentally link data elements together. In the best case, this is tedious and taxing. In the worst case, it is impossible to relate elements from one modality to another.

So now what? Well, now is the part where the visualization nerds get to cheer and applaud as the hero of this story 🦸‍♀️ visualizations 🦸‍♂️ get to swoop in and save the day! This is because…

🍢 Composite visualizations 🍢 can show different data modalities together!

What are 🍢 composite visualizations🍢? In short, composite visualizations combine multiple visualizations together into the same view. (If you want a longer/better answer, Javed and Elmqvist do a great job “Exploring the design space of composite visualization.”)

This is great for our purposes because we have three different data modalities that are each represented best by different visualizations, and we are trying to link the elements across these modalities.

What’s the catch? Even though they are powerful…

Designing composite visualizations is… (you guessed it) …hard. Here’s our approach in three simple steps.

Selected a primary data modality. Even though all data modalities are needed, specific tasks prioritize certain modalities.
Choose the best visual encoding for it. This visualization serves as the host representation.
Embed secondary data modalities as client visualizations.

Of course, it’s a bit more complicated than this, but we’ll get to that. First, let’s give an example.

Here’s one of the composite visualizations we designed (this one is my 🤗 favorite 🤗).

First, we select the tree as the primary data type and, by extension, name this composite visualization the tree-first visualization.
We encode this tree as a node-link diagram.
Then, we embed the time-series data by nesting it within the nodes of the tree and superimpose the images of cells above the nodes either automatically or on demand.

Here are the other two composite visualizations!

We call these the time-series-first visualization and the image-first visualization.

What we learned.

Ok, I have a confession. These three simple steps for constructing composite visualizations are hiding a lot of complexity. Specifically, step three is carrying a lot of weight here. In reality, this single step is a much more iterative process.

The design space of composite visualizations can be intimidatingly large. You have to choose a visual encoding for each data type AND how those encodings get combined together. The choice of one affects the other, and there will be some back and forth while exploring this space.

That said, we found that the first two steps help anchor this exploration and reduce the design space. Selecting a primary data type and prioritizing a good encoding for that data type will help ensure that tasks for that primary data type can be done effectively. With that settled, figuring out how the other pieces fit into the puzzle becomes much more manageable.

We also made a tool!

We didn’t just design these visualizations; we implemented them into a tool! It’s an excellent tool! Our collaborators like it! I like it! You can try it out yourself! You can watch a video that demonstrates it!

If you’re an extreme visualization nerd, you can even read our research paper!! The paper also goes into more detail about the design and theory pieces I talked about in this blog.

I could talk about it more, but this blog is already long enough, and I’m tired of writing it, and you are probably tired of reading it. But I promise I am actually quite proud of the tool.

Are you still here? That must mean one of two things…

You are wondering what the deal is with the aardvarks.

Well, the research paper I mentioned is titled “Aardvark: Composite Visualizations of Trees, Time-Series, and Images”. Aardvark is the name for the tool we made.

But why aardvarks?? Is it some really awesome acronym?? I bet the “v” stands for visualization!!

Sorry to burst your bubble, but it is not an acronym, I just like naming my research tools after animals (see Ferret and Loon). As to why I picked aardvarks, I think they are a cool, weird animal. 🤷

…or you want to listen to us brag about our
✨🏆✨ ~ award ~ ✨🏆✨!

We are incredibly honored and excited to share that the research paper has received a Best Paper Award at IEEE VIS 2024.

I am amped up and just a touch terrified to present to the whole conference this year in Florida! I hope to see you there! 🏖️

Ok, actually three things; you want to see the “blooper” images.

And yes, I made all of the aardvark images with AI. Specifically Adobe Firefly. These monstrosities are the result of asking for an image of an aardvark, loon, and ferret…

Devin Lange Successfully Defends Dissertation

Mon, 29 Jul 2024 11:00:00 +0000

Devin Lange successfully defended his dissertation on “Is that Right? Data Visualizations for Scientific Quality Control”. Devin was advised by Alex Lex, with Kate Isaacs, Paul Rosen, Hanspeter Pfister, and Nils Gehlenborg serving on the committee.

Devin will join Nils Gehlenborg’s Lab at Harvard Medical School as a PostDoc, while also spending some more time with us as a PostDoc. Congrats, and good luck with your next steps!