Brain-Like Intelligent Control: From Neural Nets to True Brain-Like Intelligence
Paul J. Werbos
National Science Foundation, Room 675*
Arlington, Va. 22230
This past year has seen great progress in understanding stability and developing practical applications for new learning control designs, originaly developed in the neural network area but applicable to a wider class of designs. However, there is still a big gap between the best designs implemented so far, and the kind of intelligence or higher-order decision-making capability seen in mammal brains, and required for multilevel control structures such as the management of entire factories or battlefields. This paper will briefly cite some of the prior work, but mainly summarize a new hybrid neural-AI design -- emerging from the underlying mathematics, rather than ad hoc synthesis -- which has the potential to close that gap.[1,2].
The new design draws heavily on concepts of hierarchy and temporal chunking from Artificial Intelligence (AI), and on relational representation of objects and space, and a fuzzy goal representation.
Background: Learning Control, Stability and Applications
Substantial progress has been made in the past year in developing applications, understanding stability, and improving key subsystems of learning-based controllers . Considerably more work is needed in all these directions.
In conventional discussions, it is often suggested that neurocontrol is essentially just an extension of adaptive control, which in turn is seen as the antithesis of H-infinity or robust control. (Robust control is a standard approach in control theory, often attributed to Tsypkin, reviewed in .) In actuality, this is extremely misleading.
The most popular form of robust control -- linear H¥ control, with no internal feedback loops in the controller to complicate the stability analysis -- provides high guarantees of stability, but only under strong assumptions and for limited types of plant. There are many practical situations where no controller can be designed in this family, which meet the full range of plausible parameters of the plant. For example, in building high-performance hypersonic aircraft, a recent study based on modern H-infinity design tools revealed that the range of controller parameters which could stabilize the plant when it has a center of gravity as 12 meters did not overlap at all with the set which stabilizes the plant at 11.3 meters. Even if there are ways to try to control the center of gravity, one would feel a lot safer if one could also provide a more reliable control system.
In such cases, common sense tells us that we would be safer and stabler, in practice, if we could somehow adjust the contrlller parameters to reflect what the center of gravity actually is. There ought to be some way to use current information about the center of gravity which is better (in terms of stability) than simply throwing out that information. In other words, an adaptive control strategy of some kind ought to lead to greater stability than ordinary robust control. But in practice, industry has found that this is not true for the kinds of adaptive control available to them today; as a result adaptive control is far less popular than simple robust control.
Why is it that adaptive control does not seem to have the same advantages in practice that it ought to have in theory? One reason is that the vast majority of work in adaptive control -- both classical adaptive control and neural adaptive control -- is based on minimizing some measure of tracking error one period of time ahead into the future. Substantial effort has gone into proving stability theorems for that class of design; however these stability theorems impose very strong assumptions, except in special cases where application-specific Liapunov functions happen to be known. As a practical matter, it is very easy to get into an overshoot instability when the proper direction of adaptation is not known, because the direction of change which reduces tracking error one period ahead may be opposite to what stabilizes the system many time periods ahead.
For this reason, the practical industrial applications of neural networks in control tend to involve methods which minimize error or cost many periods of time ahead into the future.
In a sense, the practical applications found in industry tend to use more advanced and brain-like designs than the ones which thereticians write about! There are two broad classes of design which are popular here: (1) methods which involve a direct simulation of future time-periods, and an explicit multiperiod optimization of some kind (e.g. Neural Model-Predictive Control, NMPC 4]); (2) methods which approximate dynamic programming.
The first class of methods is currently more popular in industrial applications, in part because it is straightforward to implement, using the details given in [4,ch.8] and [5,ch.10].
This kind of learning control has been implemented mainly with neural networks; however, it would be straightforward to use a modified, learning-based form of fuzzy logic to perform the same task. Usually, if one has the models and data required for offline learning with neural adaptive control, the upgrade to [4,ch.8] should be very easy and very effective; it provides an effective but quick fix to problems involving timing and delays, which are rampant with the simpler forms of neural adaptive control.
As an example of this approach. representatives of a major US steel company recently told me that they now use NMPC routinely to control some of their critical processes, and believe that their competitors do so also (though many details are proprietary), based in part on [5,ch.10]. Likewise, the first major success of Ford Research in using learning control to reduce emissions was based on a related approach (reviewed in ).
Ironically, however, the most consistent way to minimize costs over many time periods does not involve an explicit development of a future plan. Dynamic programming is the only exact and efficient tool available to optimize into the future in the general case, when noise or uncertainty is present. This class of methods is also far closer to what we see in the brain than is NMPC[1,2].
In actuality, the more advanced segments of the nonlinear H-infinity community
have actually been converging with the neural network community in pinpointing what is really needed, for maximum stability: i.e. control designs which are based on solving a stochastic optimization problem, using controllers which are powerful enough to function over a wider variety of possible plant parameters.
In order to cope with a wide range of possible plants, it is critical that the controller have the kind of “memory” which allows it to represent implicitly such variables as “the current center of gravity.” In H-infinity, this is typically done with “observer” components. In neurocontrol, it is done by using time-lagged recurrent networks. Either way, one can adapt the resulting complex controller by trying to minimize some measure of error or cost, in an offline mode, across a wide range of possibilities for how the plant behaves. I called this strategy“learning offline to be adaptive online” when I first proposed it in 1990. Ford Research has developed a more specific approach to implementing this strategy, which they call “multistream learning” (reviewed in ).
Given such a controller, the problem then lies in how to perform the adaptation.
Starting in the 1970’s , I have developed a family of designs -- initially implemented as neural networks -- for Approximate Dynamic Programming (ADP), which have since been used, extended and popularized by a number of researchers. In effect, the goal of these methods is to approximate the Hamilton-Jacobi-Bellman equation as accurately as possible, subject to constraints on computational effort and hardware.
As with NMPC, the main implementations have been based on neural networks so far, but this is not required by the mathematics. Likewise, both offline learning and online learning are possible.
As a practical matter, different applications require different degrees of attention to stability versus other measures of performance, In the ADP approach, this is left up to the user.
It should be noted that the “J” function approximated by these methods is also guaranteed to be a Liapunov function. Thus, to the extent that the approximation is accurate enough, these methods could be seen as a way of constructing a Liapunov function for ordinary one-period ahead adaptive control.
ADP Versus Mammalian Intelligence
By 1995, several groups of researchers had finally implemented neural network designs which met certain basic criteria for what a brain-like intelligence should look like. These designs were intended to be able to learn “any task” involving goal-directed behavior over time, without prior knowledge of the task. In technical terms, such designs have been called “model-based adaptive critics” or model-based approximate dynamic programming (ADP). They have sometimes been classified as a type of reinforcement learning system. They permit much faster learning than more primitive forms of rfeinforcement learning, at least when there are moderate or large numbers of continuous variables which need to be controlled.
The engineering benefits of such designs have become increasingly clear over the past few years. By now ten groups have successfully implemented such designs [2,8]. The strengths and weaknesses relative to conventional designs, in terms of stability, etc., are becoming better understood . In formal terms, these designs provide a capability for optimization over time in uncertain noisy nonlinear environments; that, in turn, permits systems to learn how to track a narrow desired trajectory or performance envelope, which is especially useful when trying to operate highly dynamical nonlinear plants.
In many engineering applications, neural networks are now used as part of a straightforward extension of ordinary adaptive control. In effect, matrices are simply replaced by neural networks in the same designs. In most cases, controllers are adapted so as to minimize square error in tracking at the next sampling time (t+1), based on the use of derivatives propagated through some kind of plant model, fixed or adapted. But neural net controllers using this approach have essentially the same problems with delays and nonminimum phase plants that ordinary adaptive controllers have. Using the exact same training examples, controllers designs and models, one can usually achieve better performance and stability simply by taking a different approach to training, based on multiperiod optimization. The explicit, minimum cost n-period look-ahead approach is explained in tutorial fashion in chapter 8 of  (which also contains the original papers on backpropagation from 1974 and 1981). For a more sophisticated and complete discussion, see chapter 10 of . This approach is not as popular in the journals as the ordinary adaptive approach, but it accounts for a larger share of the success in applications in industry..
In 1970 or so, when I first developed the theory behind the model-based ADP designs, I hoped that they would be enough to describe at least those aspects of higher-order intelligence which exist in the mammalian brain. This is not the same thing as explaining consciousness in the minds of humans. In order to understand consciousness in the minds of humans, one must also address issues such as symbolic reasoning, quantum computing and the soul, etc.; although I do have ideas about those subjects , they are beyond the scope of this paper. In my view, a scientific understanding of the mammal-brain level of intelligence is a prerequisite to a truly deep understanding of those more advanced levels of intelligence[2,9]. Indeed, the new design to be mentioned here does begin to provide some ideas about the deep structure underlying verbs, nouns, and adverbs, at the very least.
Nevertheless, by 1996 , it was apparent that these advanced designs still had two major deficiencies -- major gaps between the kinds of learning capability they provide, and the kind which exists in the mammalian brain.
First of all, in order to learn many difficult planning problems, such as generalized spatial navigation, it would be necessary to “fill in the boxes” in these designs with a certain type of slow but powerful neural network, the “SRN.” [10,11]. To explain how the brain also achieves high-bandwidth control of muscle movements, one then needs to assume a “two brains in one” master-slave design described in .
Second, and more seriously, in order to learn how to “see” far into the future, the underlying learning rules have to be changed, to allow some sort of “temporal chunking.” The usual critic adaptation rules train a network at time t based on information about time t+1. One can do far better by allowing the use of t+T, at times, where T is a larger time interval. But how can one do this? My concerns about this were magnified by the growing evidence from neuroscience that the basal ganglia are a crucial part of higher-order intelligence, and that they perform something very similar to task-based planning in AI.
A careful analysis of the mathematics of temporal chunking in ADP forced me into the development of an hierarchical task-based design, made up of “decision blocks.” An international patent disclosure was filed on this design on June 4, 1997, through Scientific Cybernetics, courtesy of Sam Leven. The main specification section of that disclosure is being published in . Of course, that paper mainly deals with global implementation details. (The disclosure also required 300 pages of additional papers -- mainly papers previously published -- in order to describe some of the subsystems.) In order to complement that discussion, another paper is being published very soon  which summarizes the preceding history and the intellectual strategy here in great detail. It expands on the implications for neuroscience, stressing the empirical data on the mammalian brain.
I am very grateful to Alex Meystel for drawing my attention to  , soon after the publication of . This paper made me appreciate more seriously a third limitation of the earlier model-based ADP systems -- their treatment of space. In a formal sense, this problem can be handled without changing the higher-level architecture of the model [2,12]. It is a subsystem issue, at least for the mammalian level of intelligence. (But mammals do not have to coordinate 100 quasi-independent robotic pieces of themselves spread out over an acre!) Nevertheless, the required change in neural network subsystems is very large, and has major implications for neuroscience  and engineering applications. It is particularly relevant to tasks like predicting the behavior of molecules or controlling complex networks like heat exchangers, precoolers, fuel processors, utility grids, etc.
A Few Details
Figure 1. Structure of a Decision Block
Figure 1 is a reduced version of a figure which appears in . It summarizes the structure of a
“decision block.” In this theory, the upper level of the mammalian brain (excluding the higher motivational systems) is essentially made up of a library of such decision blocks. The blocks are formally distinct, but in practice (as in the brain) they may share hidden nodes, and must share a common estimated state vector R.
The task of each decision block is to decide which lower-level decision blocks to activate. This is a crisp, discrete decision. (In , I speculated that the activation might be fuzzy instead of crisp; however, because of the problem of local minima, and the need for rational learning, this did not turn out to be promising.) For the lowest level block, the decisions are really just actions u sent to the lower brain.
Decision blocks may be thought of as “active verbs.” The arrangement is strictly sequential in nature, with only one decision block activated at any time on any level of the hierarchy.
When a decision block is activated, three kinds of additional information are also needed, to guide the actions taken inside the block: “adverb” modifiers, a fuzzy goal object g, and compressed information about the goals inherited from higher levels of the hierarchy. In , mechanisms for adapting all of these structures or networks are provided.
The fuzzy goal object mainly consists of an image of a desired state r* plus a vector of weights (indicating which components of that image vector are most important). In effect, this object provides a kind of explicit quadratic representation of a value function. The need for this kind of representation became clear only after considerable analysis of the problem of communication between decision blocks.
Crudely speaking, the benefit of fuzzy goal images may be explained as follows.
In essence, the choice of what decision to make at time t depends (directly or indirectly) on an evaluation of how good that decision would be; this evaluation may be the output as a neural network, J’(decision, state(t)) or Q(decision,state(t)), in the notation of . That network, in turn, would be trained to try to match or predict the evaluation of the outcome, J(state(t+T)), where t+T represents the completion of the decision block or action schema. In order to perform this training, using simulated experience rather than actual experience, one might use the usual strategy of sampling -- of simulating what state(t+T)
might be, based on a stochastic simulation model which inputs decision(t) and state(t) and outputs possible results. Alternatively, one might use a stochastic model which, at least in part, outputs a kind of representation of the probability distribution of state(t+T). (The R vector of the Stochastic Encoder/Decoder/Predictor (SEDP) [5,ch.13] is such a representation). Then, if the estimate of J is forced to fit a functional form which is suitabel for this purpose, it may be possible to calculate the expected J
by doing closed form integration instead of sampling. This eliminates the extra computational time and
error associated with sampling. The fuzzy goal image is one way of constraining the form of the estimate of J, consistent with the form of probability representation generated by SEDP. In the 3-brain design ,
however, additioanl fgeatures are present, to try to account for the gap between our best estimate of J and
the estimate we can realize in a fuzzy goal image; in effect, our vbalues J(t+T) are decomposed into two parts, one of which (the goal) is highly explicit, which allows for rapid replanning, and the other of which
allows for a more accurate overall representation but slower planning.
Within each decision block, neural networks must be adapted which are very similar to those in the usual model-based adaptive critics -- local critic networks, a local Model or prediction network, and a local Decision or action network. In this case, however, it is especially critical that the latter two networks be stochastic networks, rather than the usual sorts of deterministic neural network[2,12]. Two additional networks are needed in order to optimize the interface with other decision blocks.
A key aspect of using stochastic networks is that one can replicate such phenomena as “imagination” or “dreaming” or novelty-seeking behavior which is crucial to effective higher-order problem-solving and mammalian behavior [4,14].
1. P.Werbos, Learning in the Brain: An Engineering Interpretation, in K.Pribram and J.King, eds, Learning as Self-Organization, Erlbaum, 1996.
2. P.Werbos, Values, goals and utility in an engineering-based theory of mammalian intelligence, in K.Pribram, ed., Brain and Values, Erlbaum 1998.
3. P.Werbos, Neurocontrollers, in J.Webster, ed., Encyclopedia of Electronics and Electrical Engineering, Wiley, (Feb. 1998).
4. P.Werbos, The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting, Wiley, 1994.
5. D.White and D.Sofge, eds, Handbook of Intelligent Control: Neural, Fuzzy and Adaptive Approaches, Van Nostrand, 1992.
6. P.Werbos, Elastic fuzzy logic: a better fit to neurocontrol and true intelligence, J. Intelligent & Fuzzy Systems, Vol. 1, No.4, 1993.
7. P.Werbos, Advanced fore-casting for global crisis warning and models of intelligence, General Systems Yearbook, 1977 issue.
8. L.Dolmatova and P.Werbos, Traps and tricks in standard benchmark problems for neurocontrol. In A. Meystel, ed., Proc. of 1997 Int’l Conf. on Intelligent Systems and Semiotics, NIST Special Publication 918. Washington D.C.: GPO, 1997.
9. P.Werbos, Optimization: A Foundation for understanding consciousness. In D. Levine & W. Elsberry eds Optimality in Bio-logical and Artificial Networks?, Erlbaum, 1997.
10. P.Werbos & X.Z.Pang, Generalized maze navigation: SRN critics solve what feed-forward or Hebbian nets cannot. Proc. Conf. Systems, Man and Cybernetics (SMC) (Beijing), IEEE, 1996. (An earlier version appeared in WCNN96.)
11. X.Pang & P.Werbos, Neural net-work design for J function approximation in dynamic programming,Math. Modelling and Scientific Computing (a Principia Scientia journal), Vol. 5, NO.2/3, 1996. Available as adap-org 9806001 from xxx.lanl.gov/form using the “Other Groups” field set to “nlin-sys.”
12. P.Werbos, A Brain-Like Design To Learn Optimal Decision Strategies in Complex Environments, in M.Karny, K.Warwick and V. Kurkova eds Dealing with Complexity: A Neural Networks Approach. Springer, London, 1998.
13. J.Albus, Outline of Intelligence, IEEE Trans. Systems, Man and Cybernetics, Vol.21, No.2, 1991.
14. D.S.Levine and S.J.Leven, eds, Motivation, Emotion, and Goal Direction in Neural Networks, Erlbaum, 1992.
* The views herein are those of the author, not those of NSF, but the paper was written on NSF time and may therefore be freely copied subject to proper citation.