Otdfbt

I was reading the transcripts of a seminar by Dr. Richard W. Hamming. In this, Hamming goes on to say the following statements:

Explaining why one guy was successful while the others weren't as much successful.

I have never heard the names of any of the other fellows at that table
mentioned in science and scientific circles. They were unable to ask
themselves, "What are the important problems in my field?" If you
do not work on an important problem, it's unlikely you'll do important
work. But the average scientist does routine safe work almost all
the time and so he (or she) doesn't produce much. It's that simple. If
you want to do great work, you clearly must work on important
problems, and you should have an idea. Most great scientists know
many important problems. They have something between 10 and 20
important problems for which they are looking for an attack. And when
they see a new idea come up, one hears them say "Well that bears on
this problem."

Now my question is: If I ask myself the question, What are the important problems in my field?, how should I expect to get an answer to this. Is it by deeply diving into literature and finding loopholes that exist. And even if I do find something worth working upon, how do I know it's an important problem of the field or a trivial subset of some other large problem?

I am sorry but I am unable to get myself a clear picture of what exactly is meant here. Will anyone please help me out with ideas.

Many thanks.

What are the important problems in my field?, how should I expect to get an answer to this. Is it by deeply diving into literature and finding loopholes that exist. And even if I do find something worth working upon, how do I know it's an important problem of the field or a trivial subset of some other large problem?

I would interpret this statement not so much as trying to look for "loopholes" in the literature, more as trying to identify what the core problems are that are holding back further advancement in your field, and what would need to change in order for them to become tractable. This requires you to deeply understand the existing work, so as to learn what exactly needs to change for your field to really jump forward.

However, I should warn you - Hamming’s approach is certainly a high-risk-high-reward one. If you implement this, you will almost certainly have to invest quite some work detailling and understanding a large number of difficult problems that you will in all likelihood make no progress on during your lifetime. Really important problems invariably don't have a straight-forward solution (or, in many cases, no solution at all), so even if you bring yourself into the best position to attack them in the right circumstances, there is little guarantee that these circumstances ever come to pass (or that you are the first to recognize if some recent development in the field makes one of "your" open problems tractable). That said, if you manage to address even one really big, important problem, your impact will indeed be higher than what you can achieve in decades of incremental research.

Not everyone is a Richard Hamming or a John Tukey. They did what they did because they were brilliant. It's not like everyone can become them just by deciding to work on something different.

My advice is to keep a notebook and write down ideas. Occasionally do things like putting them in Excel and organizing them into categories (including multiple tags).

Personally, I WOULD find some safe work to do. That you know delivers results and that you can spend time on and get rewards from. Try to at least work up to, and around more fundamental ideas. But it is also good to "grind your pigments", not just jump to painting Mona Lisas.

The important problems in each field are different, of course. Some of them are well known and well publicized. In CS, "Does P = NP" is one of those problems that every advanced student is introduced to. The same will be true of such problems in other fields. However, one of the reasons that such problems are important and well known is that they are very hard and that lots of people have tried and failed to solve them. There is normally a rich literature on such long-standing problems. But you aren't likely to solve them yourself. Some may even not be solvable at all.

Other problems are important, but unknown. They won't be known to be important until they are solved and the solutions are found to open up new lines of thought and inquiry. These are often solved purely by serendipity. Someone has an "interesting" thought and explores it. That leads to other interesting thoughts. If this work becomes known it may be post facto treated as important.

But a solo researcher is pretty unlikely to be the one to solve such problems, though it can happen.

Collaboration is the key

Important problems are most often solved by people who are not only able, and well trained, but also connected to other such people. In a research group, whether co-located or not, there are a tremendous number of ideas floating around and discussed. Some of these problems are big problems and some of them are just "interesting looking" areas of inquiry that haven't yet been mined for results. If you can get connected, somehow, to such a group then you will learn about those big, but not well-known, problems that might be worth attacking.

This is why research seminars at large departments are so valuable. People with a somewhat different viewpoint than your own can make suggestions, some of which can bear fruit. Doctoral dissertations often come from such seminars and some of them are important work on important problems, whether they provide a complete solution or not.

So, if there is advice to give here, my advice would be to look, first, for "interesting ideas" not important problems. Many of the important problems aren't known to be such so looking for them directly isn't fruitful. But find a group to work with and look at a lot of ideas. Some of them will, hopefully, turn out to be interesting. Hamming didn't work alone, of course.

And yes, what is "interesting" varies by field and with time.

While I think guest's answer is relevant, here is a counterpoint.

It is true that not everyone will be able to solve outstanding problems in one's field. However, (imo) a recipe for bad science is to specialize to the extreme and keep working on a narrow set of problems, interesting only to a tiny community that doesn't exchange ideas with the rest of the field.

Here's how to know if a problem is important : talk with other people in your field, see what interests them. Everyone has their own backgrounds and their own interests, but the more attention a problem draws, the more important it is.

The amount of literature is also relevant: if there are a lot of publications from well-established people all around the world on a certain topic, then it's interesting (at least to people from your field). If the only people publishing papers about it are 2 guys in an obscure university, who are only referenced by each other, and who wrote 15 papers on the same topic all published in the same journal, it's probably not a serious topic.

So in conclusion, for most people it may be unrealistic to want to only work on major problems. BUT it doesn't mean that you shouldn't try to tackle (reasonnably easier) problems that are of interest to your community, instead of focusing too much on "what you know you can do".