What are the essential features of a theory for the foundations of physics? What demands should a researcher or physicist make of a theory for it to be useful?
Before we start, we should mention that although the word "theory" has specific meaning, we use it here in a very broad sense to cover hypotheses, ideas, schemes, models, or 'theories-proper'. It's also worth noting that we are writing about the discovery of new underlying foundations for physics, rather than any (foolish) attempt to replace the accepted legacy of physics with a new regime.
As we've written elsewhere, the pursuit of good foundations is usually driven by a quest for knowledge, the prospect of new discoveries, and the desire to defeat a bizarre puzzle where the answers cannot found at the back of the book!
If we have good foundations for a subject area, then it's likely that we'll have a simple set of tools and rules that allows us to understand the body of the subject with little other supplemental material. Those rules are likely to apply throughout the subject, and when the foundations are not quite correct, we can learn from the exceptions to the theory, and refine it accordingly. A long-refined theory is likely to have missed some important foundations from the very beginning that are not as equally simple as the original elements of the theory, and so are usually difficult to assimilate, whereas a revolutionary theory is likely to use new and simpler ideas, with more of the foundational terms being unified or equivalent in some way, but the down-side is that the ideas are likely to be incomprehensible to the majority - even those qualified to understand. Good theories often need re-interpreting to be understood by a wider readership.
In a unified theory, there are symmetries: aspects of a state that will not change under transformation, while other aspects do change. The ideal theory will be constructed almost entirely from similar symmetrical terms at a fundamental level, and these symmetries have implications for the whole subject, finding expression in the description of a reality, and having consequences at all scales.
Theoretical physicists need to be creative to produce original ideas. It often requires deep knowledge and insight into the subject area, a very associative mindset (where many connections between many things are considered, even if most are rejected), and a diverse imagination that allows the physicist to see things differently, so that new ways can be found to create principles and structures that underpin the subject. Persistence and determination also play a part, especially considering that a 'discovery' is the first successful communication of an given idea, and to be first, you have to have made more progress than your predecessors. There is also a creative aspect to the way that scientists present their material to make it successful, either in their writing style, the approach and structure of their publications, the way they negotiate and communicate with like-minded researchers, and the infectious enthusiasm they generate when lecturing to colleagues or communicating with the public.
On a more formal level, a theory or hypothesis is prepared in a process of creative design. There will be some objectives (otherwise there would be no project!), and some rules or design principles to follow during the creative process, to provide technical steering. As with the computer systems 'development life cycle':
Eventually, a good theory will be released in a complete form, or it may be 'beta-tested' to allow others to test-drive it. Some theories are good enough that they are kept for proprietary advantage, or are (more responsibly) released for others to use.
In the creative process, a designer may be presented with (or invent) options to resolve a requirement. For example, there might be a need to find a means for fields to change the position of a particle; there might be several ways that terms for fields to affect the relative states of conserved entities. 'Design decisions' will need to be made to choose between available options. Such design decisions are an intrinsic part of the creative process, and exploration of such options should be encouraged, so long as the designer is mindful of the need to converge the ideas with reality at some point, without 'losing the plot'. There should also be an awareness that choosing a perspective might colour or polarise any dependent ideas, and that an alternative choice is not necessarily the wrong one; dualities often present a choice of equally correct perspectives.
We could easily quote some typical guidelines for paper submission, as they would of course apply here:
Applying some of these to the construction of a theory of foundational physics, a useful theory might do the following (non-exhaustive list):
Philosophy and formal mathematics are both strict about the traceability of ideas, and notions of truth and proof. Perhaps we should adopt the same rigour with all the entities, interfaces and processes that we design, by defining their origins and relationships with other phenomena, such that the whole body of work can be built from its lowest foundations to its highest hierarchical levels by the controlled introduction of separate principles and assumptions. The work then becomes traceable, and if an aspect of the work is found to be flawed or inadequate, the process of auditing, and then invalidating or refining the affected areas becomes relatively simple.
Ideas must relate to reality, and be reconcilable with experiment and observation (macro-extension). Ideas must also be reconcilable to proposed foundational principles (micro-extension). Understanding the scope and relations between phenomena will help identify incomplete mappings, a need for transformation or context, or enable new discoveries.
Complete vertical traceability needn't be demonstrated in publications; it may just be a handy tool to help with the design process.
There must be an explanation of how matter exists, rather than an assumption that it does exist. This means digging deeper than the notion of a physical entity like a particle or a wave, to speculate its components. It is healthy, rather than foolish, to endeavour to break a seemingly fundamental fermion (quark or lepton) or boson into smaller parts. These parts might not exist in the same way that we expect fermions to, and it takes some imagination to find a reasonable way that component states can exist 'behind the scenes' that may, in some circumstances, produce the phenomena that we currently recognise as fundamental (even if we can't observe them directly). This approach has produced revolutionary ideas throughout history.
To help points 1 and 2, concepts and phenomena should be placed in a hierarchy. The levels should be states or expressions of the model, and steps between levels should be processes; both of which may have mathematical description. All levels must be implicit and self-generative from lower levels, without need of external ‘handle-turning’ processes.
For any given observation, it should then be possible to explain the epistemology of how the measurement becomes macroscopically quantifiable, and it should be possible to explain a traceable path from foundations, through the hierarchy of all the expressions and processes that contribute to the observation. This necessarily includes perspectives ('pictures') on the fundamental data, choosing appropriate limits in appropriate variables, understanding the linearity and complexity of a model, taking products and projections as appropriate to the model, and as a bonus should identify areas where information is discarded, mixed, or transformed by the mathematical or physical model. Most importantly, numeric predictions are testable, and the model can be evaluated on the accuracy of the predictions.
The distinction should be made between (a) abstractions and (b) terms that are proposed to exist as entities or states. The transition to abstraction can then be seen as a point for further improvement (without precluding useful abstractions).
Where extra terms or phenomena are added, they must not be arbitrary, but must instead be explained in terms of the existing foundational terms. In other words, a new term must be either a new symmetry (that yields new discoveries in new expressions), or a new phenomena (which is described in terms of existing symmetries, and has other dual aspects or sibling terms).
Of course, we should never let formality get in the way of a good idea; a sketch on an envelope could generate as much excitement as an edition of a listed journal of peer-reviewed conference proceedings. The formality helps the communication of an idea, protects the assertion of the scientist as the work's author, is a form that is familiar and comfortable to digest, and with some editorial authority that it is 'good science'.
Beware the misguided individual who continues to evangelise poor work as good work, and who over-inflates the importance of their ideas: the 'crackpot physicist'. The problem with such cases is usually a lack of strength in depth of knowledge in physics (rather than a lack of formal education), and an inappropriate sense of associativity between physical phenomena. As in nearly all aspects of life, significant improvements can be made by listening to the advice of experts (if the author is lucky enough to receive it), admitting errors, acknowledging weaknesses, and being willing to revise or discard flawed aspects without being too precious about them.
Also beware people who censor ideas that they cannot understand, or who discredit ideas only because they devalue other areas of research (particularly their own). Such behaviour is not in the scientific interest, and it is the duty of physics (as a body of researchers) to be open to inspection and to be open to alternative ideas that have potential to improve the field. Breakthroughs in foundational physics are never convenient at the time they occur, but as years (and old creeds) pass, revolutionary ideas are developed into good theories or principles, and eventually become part of the curriculum.
As mentioned above, 'marketing' can have a critical bearing on the success of an idea: poor presentation can cause a good idea to fail, and good presentation of a poor idea can mislead and waste time. Throughout the whole process – from conception and refinement of ideas, through to the style of their communication, collaborative developments, and the willingness to accept ideas – a creative process is at work.