Introduction to Neural Networks and Learning Machines

  • A neural network is a massively parallel distributed processor made up of simple processing units that has a natural propensity for storing experiential knowledge and making it available for us.
  • Brain is a highly complex, nonlinear and parallel computer.
  • Brain is able to accomplish perceptual recognition tasks in 100-200 ms whereas tasks of much lesser complexity take a great deal longer on a powerful computer.
  • Much of the development of human brain taking place during the first two years of birth! But the development continues well beyond this stage.
  • Plasticity permits the developing nervous system to adapt to its surrounding environment.
  • Neural Network is a machine that is designed to model the way in which the brain performs a particular task or function of interest.
  • Learning algorithm is the function of which to modify the synaptic weights of the network in an orderly fashion to attain a desired design objective.
  • Properties and capabilities on NN:
    • Nonlinearity.
    • Input-Output Mapping.
    • Adaptivity.
      • The principal time constants of the system should be long enough for the system to ignore spurious disturbances, and yet short enough to respond to meaningful changes in the environment. This is the problem on stability-plasticity dilemma.
    • Evidential Response.
      • Supplying each decision with confidence factor.
    • Contextual Information.
      • Every neuron in the network is affected by the global activity of all other neurons in the network.
    • Fault Tolerance.
    • VLSI Implementation.
    • Uniformity of Analysis and Design.
    • Neurobiological Analogy.
  • It’s estimated that there are approximately 10 billion neuron in human cortex and 6o0 trillion synaptic.
  • Synapses or nerve endings are elementary structural and functional units that mediate the interconnections between neurons.
  • Adaptivity in human brain is made by:
    • Creation of new synaptic connections between neurons or,
    • Modification of existing synapses.
  • ANN we are presently able to design is just as primitive compared with the local circuits and the interregional circuits of the brain.
  • Types of activation function:
    • Threshold Function (Heaviside Function).
    • Sigmoid Function.
  • See page 46: mathematical definition of neural network (as a directed graph) and 4 properties of it.
  • See page 47: partially complete directed graph (architectural graph) and its properties.
  • The manner in which the neurons of a NN are structured is intimately linked with the learning algorithm used to train the network.
  • Networks Architecture:
    • Single-Layer Feedforward Networks.
    • Multilayer Feedforward Networks.
      • By adding one or more hidden layers the network is enabled to extract higher-order statistics from its input.
      • We’ve two types of connected networks: fully connected and partially connected.
    • Recurrent Networks.
      • Self-feedback refers to a situation where the output of a neuron is fed back into its own input.
  • Knowledge refers to stored information or models used by a person or machine to interpret, predict, and appropriately respond to the outside world.
  • Characteristics of Knowledge Representation:
    • What information is actually made explicit?
    • How the information is physically encoded for subsequent use?
  • 55-See differences between pattern classifiers and neural networks in page.
  • Knowledge representation of the surrounding system is environment is defined by the values taken by the free parameters (i.e. synapses and bias) of the network.
  • Knowledge Representation Rules:
    • Similar inputs from similar classes should usually produce similar representations inside the networks and should therefore be classified as belonging to the same class.
    • Items to be categorized as separate classes should be given widely different representations in network.
    • If a particular feature is important, then there should be a large number of neurons involved in the representation of that item in the network.
    • Prior information and invariances should be built into the design of a neural network whenever they are available. So as to simplify the network design by not having to learn them.
  • To find similarity for deterministic terms we use Euclidian distance. For stochastic terms we use Mahalanobis distance.
  • Specialized Structured Neural Networks are desired for the following reasons:
    • Having smaller number of free parameters. This lead to small number of training, network learns fast and often generalizes better.
    • The rate of information transmission through a specialized network (i.e. the network throughput) is accelerated.
    • The cost is reduced because its smaller size relative to fully connected network.
  • Ad hoc techniques to build prior information into neural network:
    • Restricting the network architecture, this is achieved through the use of local connections known as receptive fields.
    • Constraining the choice of synaptic weights, which is implemented through the use of weight sharing.
  • Receptive field of a neuron is defined as the region of the input field over which the incoming stimuli can influence the output signal produced by the neuron.
  • Techniques for rendering classifier-type neural network invariant to transformations:
    • Invariance by structure:
      • Synaptic connections between the neurons of the network are created so that transformed versions if the same input are forced to produce the same output. (i.e image center rotation)
    • Invariance by training:
      • Ability to recognize an object from different perspectives using several aspect views.
      • Disadvantages from engineering aspect:
        • Probability of misclassification.
        • High computation demand (especially with high features dimensions)
    • Invariant feature space:
      • This technique relies on the ability of extracting features that characterize the essential information content of an input data set and that are invariant to transformations.
      • Advantages of using this technique:
        • Reduced number of features.
        • Requirements of the design are relaxed
        • Invariance for all objects with respect to known transformations is assured.
  • Learning Paradigms:
    • Supervised Learning.
    • Unsupervised Learning.
    • Reinforcement Learning.
  • Learning Tasks:
    • Pattern Association.
      • Associative memory is a brain like distributed memory that learns by association.
      • Association forms:
        • Autoassociation (Unsupervised).
        • Heteroassociation (Supervised).
      • Phases of associative memory:
        • Storage phase.
        • Recall phase.
      • Challenge here is to make the storage capacity q (expressed as a percentage of the total number N neurons used to constructs the network) as large as possible.
    • Pattern Recognition.
      • Pattern recognition is the process of receiving a pattern/signal and assign it to one of prescribed number of classes.
      • Forms of pattern recognition machines using neural networks:
        • Machine is constructed from feature extractor and supervised classification.
          • Feature extractor applies dimensionality reduction (i.e. data compression).
        • Machine is constructed from Feedforward network using supervised learning algorithm.
          • The task of feature extraction is performed by the computational units in the hidden layers of the network.
    • Function Approximation.
      • Given a set of labeled examples, the requirement is to design a neural network that approximates the unknown function f(.) such that the function F(.) describes input-output mapping actually realized by the network, is close enough to f(.) in Euclidean sense over all inputs (i.e. for all x)
      • Ability of a NN to approximate an unknown input-output mapping is characterized by:
        • System identification: ability to identify key patterns.
        • Inverse modeling:
    • Control.
      • Primary objective of the controller is to supply appropriate inputs to the plant to make its output y track the referenced signal d. In other words, the controller has to invert the plant’s input-output behavior.
      • Approaches for accounting k, j:
        • Indirect Learning.
        • Direct Learning.
    • Beamforming.
      • Beamforming is used to distinguish between the spatial properties of the target and background noise. The device used to do the Beamforming is called a beamformer.
      • Task of Beamforming is complicated according to two factors:
        • Target signal originates from an unknown direction.
        • There is no prior information available on the inferring signals.
  • Key terms: key pattern, memorized pattern, error in recall, memoryless MIMO system, neuro-beamformer, attentional neurocomputers, semisupervised learning.
  • Questions:
    • What’s linear adaptive filter theory?
    • What’s tabula rasa learning?
    • Page 33 line 19. What’s meant by this paragraph?
    • Needs more discussion about “Uniformity of Analysis and Design”.
    • Discussion about the 2 examples in page 35.
    • Last paragraph in page 37.
    • What’s logistic function?
    • 48-What we mean by dynamic system?
    • 48-Why A and B are operators? And what’s the resulted from this?
    • 48-What’s the difference between A and ?
    • 48-What we mean by non-commutative?
    • 48-What are the properties on non-commutative operators?
    • 49-Explanation of equation 19, 20?
    • 49-What’s binomial expansion?
    • 49-Explanation of 2 cases in bottom.
    • 53-Last paragraph
    • 59- Ad hoc techniques to build prior information into neural network.
    • 60-What are the differences between convolution network and usual networks?
    • 61-What’s meant by occlusion?
    • 65-2nd portion of page until unsupervised learning.
    • 66-Reinforcement learning paragraph
    • 68-What is meant by space of dimensionality?
    • 73-Is system identification is done using control task?
    • 73-In equation 37 how he’ll get the differentiation of a constant?
    • 73-What is meant by element of a planet?
    • 73-What’s the problem of j, k?
    • 73-What’s direct and indirect learning?
    • 74-Discussion on diagram of generalized sidelobe canceller.

2 thoughts on “Introduction to Neural Networks and Learning Machines

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s