Genetic Programming

Positional Effect of Crossover and Mutation in Grammatical Evolution

by Tom Castle

An often-mentioned issue with Grammatical Evolution is that a small change in the genotype, through mutation or... more An often-mentioned issue with Grammatical Evolution is that a small change in the genotype, through mutation or crossover, may completely change the meaning of all of the following genes. This paper analyses the crossover and mutation operations in GE, in particular examining the constructive or destructive nature of these operations when occurring at points throughout a genotype. The results we present show some strong support for the idea that events occurring at the first positions of a genotype are indeed more destructive, but also indicate that they may be the most constructive crossover and mutation points too. We also demonstrate the sensitivity of this work to the precise definition of what is constructive/destructive.

Evolving High-Level Imperative Program Trees with Strongly Formed Genetic Programming

by Tom Castle

We present a set of extensions to Montana's popular Strongly Typed Genetic Programming system that introduce... more We present a set of extensions to Montana's popular Strongly Typed Genetic Programming system that introduce constraints on the structure of program trees. It is demonstrated that these constraints can be used to evolve programs with a naturally imperative structure, using common high-level imperative language constructs such as loops. A set of three problems including factorial and the general even-n-parity problem are used to test the system. Experimental results are presented which show success rates and required computational effort that compare favourably against other systems on these problems, while providing support for this imperative structure.

Evolving Program Trees with Limited Scope Variable Declarations

by Tom Castle

Variables are a fundamental component of computer programs. However, rarely has the construction of new variables been... more Variables are a fundamental component of computer programs. However, rarely has the construction of new variables been left to the evolutionary process of a tree-based Genetic Programming system. We present a series of modifications to an existing GP approach to allow the evolution of high-level imperative programs with limited scope variables. We make use of several new program constructs made possible by the modifications and experimentally compare their use. Our results suggest the impact of variable declarations is problem dependent, but can potentially improve performance. It is proposed that the use of variable declarations can reduce the degree of insight required into potential solutions.

EpochX: Genetic Programming in Java with Statistics and Event Monitoring

by Tom Castle

EpochX is a Genetic Programming (GP) framework written in Java. It allows the creation of tree-based and grammar-based... more EpochX is a Genetic Programming (GP) framework written in Java. It allows the creation of tree-based and grammar-based GP systems. It has been created to reflect typical ways in which Java programmers work, for example, borrowing from the Java event model and taking inspiration from the Java collections framework. This paper presents EpochX in general, and gives particular attention to the event model and the statistics analysis framework.

Integrating human knowledge within a hybrid clustering-classification scheme for detecting patterns within large movement data sets

by Benjamin Fowler

Co-authored with René Enguehard, Orland Hoeber, Rodolphe Devillers, and Wolfgang Banzhaf
Published in AGILE 2012

The visual analysis of large movement data sets can be a challenging task. This study proposes an approach for... more The visual analysis of large movement data sets can be a challenging task. This study proposes an approach for identifying interesting movement patterns that combines human knowledge and decision making with a hybrid clustering-classification method. Rather than performing an unsupervised clustering of the entire data set, a stratified random sample of the full data set is used to identify initial clusters that are verified and labelled by the analyst, and then used as input patterns for classifying the remainder of the data set using an iterative genetic program. Classifications suggested after each iteration are presented to the analyst for refinement based on their knowledge and experience. A geovisual analytics environment is provided to both show the outcomes of the clustering and classification, and to obtain the analyst’s input during the hybrid clustering-classification process. Our approach allows data to be classified without a priori specification of classification patterns. Instead, the process takes advantage of human decision making within the automatic analysis of the data. The approach was tested with fishing vessel movement data in Eastern Canada.

Money in Trees: How Memes, Trees, and Isolation Can Optimize Financial Portfolios

by Carlos R. B. Azevedo

Evolving Object Oriented Agent Programs In Robocup Domain

by Julian Olarte

Towards the Automatic Design of Decision Tree Induction Algorithms

by Rodrigo Barros

Co-authored with Marcio Basgalupp and Andre Carvalho and Alex Freitas, published in the proceedings of the 2011 GECCO, 2011.

Decision tree induction is one of the most employed methods to extract knowledge from data, since the representation... more Decision tree induction is one of the most employed methods to extract knowledge from data, since the representation of knowledge is very intuitive and easily understandable by humans. The most successful strategy for inducing decision trees, the greedy top-down approach, has been continuously improved by researchers over the years. This work, following recent breakthroughs in the automatic design of machine learning algorithms, proposes two different approaches for automatically generating generic decision tree induction algorithms. Both approaches are based on the evolutionary algorithms paradigm, which improves solutions based on metaphors of biological processes. We also propose guidelines to design interesting fitness functions for these evolutionary algorithms, which take into account the requirements and needs of the end-user.

Evolutionary Algorithms

by Shiburaj Pappu

Special Topic Seminar on Evolutionary Algorithm

Evolutionary algorithm (EA) is a generic population-based metaheuristic optimization algorithm. An EA uses some... more Evolutionary algorithm (EA) is a generic population-based metaheuristic optimization algorithm. An EA uses some mechanisms inspired by biological evolution: reproduction, mutation, recombination, and selection. Candidate solutions to the optimization problem play the role of individuals in a population, and the tness function determines the environment within which the solutions "live" . Evolution of the population then takes place after the repeated application of the above operators.
This description is deliberately based on a unifying view resenting a general scheme that forms the common basis of all volutionary Algorithm variants. The main components of EAs are discussed, explaining their role and related issues of terminology. Further on we discuss general issues for EAs concerning their working. Finally, we put EAs into a broader context and explain their relation with other global optimisation techniques.

Genetic programming metamodel for rotating beams

by Ranjan Ganguli

Automatic Induction of Projection Pursuit Indices

by Eduardo Rodriguez Martinez

Published in IEEE TNN, Aug, 2010

Projection techniques are frequently used as the principal means for the implementation of feature extraction and... more Projection techniques are frequently used as the principal means for the implementation of feature extraction and dimensionality reduction for machine learning applications. A well established and broad class of such projection techniques is the projection pursuit (PP). Its core design parameter is a projection index, which is the driving force in obtaining the transformation function via optimization, and represents in an explicit or implicit way the user's perception of the useful information contained within the datasets. This paper seeks to address the problem related to the design of PP index functions for the linear feature extraction case. We achieve this using an evolutionary search framework, capable of building new indices to fit the properties of the available datasets. The high expressive power of this framework is sustained by a rich set of function primitives. The performance of several PP indices previously proposed by human experts is compared with these automatically generated indices for the task of classification, and results show a decrease in the classification errors.

Learning comprehensible classification rules from gene expression data using genetic programming and biological ontologies

by Francisco Prosdocimi

We consider the problem of how to use automated techniques to learn simple and compact classification rules from... more We consider the problem of how to use automated techniques to learn simple and compact classification rules from microarray gene expression data. Our approach employs the traditional “genetic programming” (GP) algorithm as a supervised categorization technique, but rather than applying GP to gene expression vectors directly, it applies GP to “enhanced feature vectors” obtained by preprocessing the gene expression data using the Gene Ontology and PIR ontologies. On the two datasets considered, this “GP + enhanced feature vectors” combination succeeds in producing compact and simple classification models with near-optimal classification accuracy. For sake of comparison, we also give results from the combination of support vector machine classification and enhanced feature vectors on the same datasets.

Automating the Packing Heuristic Design Process with Genetic Programming

by Matthew Hyde

Evolutionary Computation (Accepted, To Appear)

The literature shows that one, two and three dimensional bin packing and knapsack packing are difficult problems in... more The literature shows that one, two and three dimensional bin packing and knapsack packing are difficult problems in Operational Research. Many techniques, including exact, heuristic, and metaheuristic approaches, have been investigated to solve these problems and it is often not clear which method to use when presented with a new instance. This paper presents an approach which is motivated by the goal of building computer systems which can design heuristic methods. The overall aim is to explore the possibilities for automating the heuristic design process.

We present a genetic programming system to automatically generate a good quality heuristic for each instance. It is not necessary to change the methodology depending on the problem type (one, two or three dimensional knapsack and bin packing problems), and it therefore has a level of generality unmatched by other systems in the literature. We carry out an extensive suite of experiments and compare with the best human designed heuristics in the literature. Note that our heuristic design methodology uses the same parameters for all the experiments.

The contribution of this paper is to present a more general packing methodology than those currently available, and to show that, by using this methodology, it is possible for a computer system to design heuristics which are competitive with the human designed heuristics from the literature. This represents the first packing algorithm in the literature able to claim human competitive results in such a wide variety of packing domains.

Grammatical Evolution of Local Search Heuristics

by Matthew Hyde

IEEE Transactions on Evolutionary Computation (Accepted, To Appear)

Genetic programming approaches have been employed in the literature to automatically design constructive heuristics... more Genetic programming approaches have been employed in the literature to automatically design constructive heuristics for cutting and packing problems. These heuristics obtain results superior to human created constructive heuristics, but they do not generally obtain results of the same quality as local search heuristics, which start from an initial solution and iteratively improve it. If local search heuristics can be successfully designed through evolution, in addition to a constructive heuristic which initialises the solution, then the quality of results which can be obtained by automatically generated algorithms can be significantly improved. This paper presents a grammatical evolution methodology which automatically designs good quality local search heuristics that maintain their performance on new problem instances.

Genetic programming of Fuzzy logic production rules with application to financial trading

by Andrew Edmonds

Neural networks in Financial Engineering, pp. 179-188, World Scientific, Nov. 1995


This was the first paper on GP for evolving financial trading systems and the first paper linking GP & Fuzzy Logic.

John Koza7 has demonstrated that a form of machine learning can be constructed
by using the techniques of Genetic... more John Koza7 has demonstrated that a form of machine learning can be constructed
by using the techniques of Genetic Programming using LISP statements. We
describe here an extension to this principle using Fuzzy Logic sets and
operations instead of LISP expressions. We show that Genetic programming can
be used to generate trees of fuzzy logic statements, the evaluation of which
optimise some external process, in our example financial trading. We also show
that these trees can be simply converted to natural language rules, and that these
rules are easily comprehended by a lay audience. This clarity of internal
function can be compared to “Black Box” non-parametric modelling techniques
such as Neural Networks. We then show that even with minimal data
preparation the technique produces rules with good out of sample performance
on a range of different financial instruments.

Exploring Hyper-heuristic Methodologies with Genetic Programming

by Matthew Hyde

In Computational Intelligence: Collaboration, Fusion and Emergence (eds. Mumford C. L. and Jain L. C.). Springer-Verlag. pp. 177--201

Hyper-heuristics represent a novel search methodology that is motivated by the goal of automating the process of... more Hyper-heuristics represent a novel search methodology that is motivated by the goal of automating the process of selecting or combining simpler heuristics in order to solve hard computational search problems. An extension of the original hyper-heuristic idea is to generate new heuristics which are not currently known. These approaches operate on a search space of heuristics rather than directly on a search space of solutions to the underlying problem which is the case with most meta-heuristics implementations. In the majority of hyper-heuristic studies so far, a framework is provided with a set of human designed heuristics, taken from the literature, and with good measures of performance in practice. A less well studied approach aims to generate new heuristics from a set of potential heuristic components. The purpose of this chapter is to discuss this class of hyper-heuristics, in which Genetic Programming is the most widely used methodology. A detailed discussion is presented including the steps needed to apply this technique, some representative case studies, a literature review of related work, and a discussion of relevant issues. Our aim is to convey the exciting potential of this innovative approach for automating the heuristic design process.

Evolving Bin Packing Heuristics with Genetic Programming

by Matthew Hyde

The bin-packing problem is a well known NP-Hard optimisation problem, and, over the years, many heuristics have been... more The bin-packing problem is a well known NP-Hard optimisation problem, and, over the years, many heuristics have been developed to generate good quality solutions. This paper outlines a genetic programming system which evolves a heuristic that decides whether to put a piece in a bin when presented with the sum of the pieces already in the bin and the size of the piece that is about to be packed. This heuristic operates in a fixed framework that iterates through the open bins, applying the heuristic to each one, before deciding which bin to use. The best evolved programs emulate the functionality of the human designed ‘first-fit’ heuristic. Thus, the contribution of this paper is to demonstrate that genetic programming can be employed to automatically evolve bin packing heuristics which are the same as high quality heuristics which have been designed by humans.

A Genetic Programming Hyper-Heuristic Approach to Automated Packing

by Matthew Hyde

This thesis presents a programme of research which investigated a genetic programming hyper-heuristic methodology to... more This thesis presents a programme of research which investigated a genetic programming hyper-heuristic methodology to automate the heuristic design process for one, two and three dimensional packing problems.

Traditionally, heuristic search methodologies operate on a space of potential solutions to a problem. In contrast, a hyper-heuristic is a heuristic which searches a space of heuristics, rather than a solution space directly. The majority of hyper-heuristic research papers, so far, have involved selecting a heuristic, or sequence of heuristics, from a set pre-defined by the practitioner. Less well studied are hyper-heuristics which can create new heuristics, from a set of potential components.

This thesis presents a genetic programming hyper-heuristic which makes it possible to automatically generate heuristics for a wide variety of packing problems. The genetic programming algorithm creates heuristics by intelligently combining components. The evolved heuristics are shown to be highly competitive with human created heuristics. The methodology is first applied to one dimensional bin packing, where the evolved heuristics are analysed to determine their quality, specialisation, robustness, and scalability. Importantly, it is shown that these heuristics are able to be reused on unseen problems. The methodology is then applied to the two dimensional packing problem to determine if automatic heuristic generation is possible for this domain. The three dimensional bin packing and knapsack problems are then addressed. It is shown that the genetic programming hyper-heuristic methodology can evolve human competitive heuristics, for the one, two, and three dimensional cases of both of these problems. No change of parameters or code is required between runs. This represents the first packing algorithm in the literature able to claim human competitive results in such a wide variety of packing domains.

Providing a Memory Mechanism to Enhance the Evolutionary Design of Heuristics

by Matthew Hyde

Genetic programming approaches have previously been employed in the literature to evolve heuristics for various... more Genetic programming approaches have previously been employed in the literature to evolve heuristics for various combinatorial optimisation problems. This paper presents a hyper-heuristic genetic programming methodology to evolve more sophisticated one dimensional bin packing heuristics than have been evolved previously. The heuristics have access to a memory, which allows them to make decisions with some knowledge of their potential future impact. In contrast to previously evolved heuristics for this problem, we show that these heuristics evolve to draw upon this memory in order to facilitate better planning, and improved packings. This fundamental difference enables an evolved heuristic to represent a dynamic packing strategy rather than a fixed packing strategy. A heuristic can change its behaviour depending on the characteristics of the pieces it has seen before, because it has evolved to draw upon its experience.

Automatic Heuristic Generation With Genetic Programming: Evolving a Jack-of-All-Trades or a Master of One

by Matthew Hyde

It is possible to argue that online bin packing heuristics should be evaluated by using metrics based on their... more It is possible to argue that online bin packing heuristics should be evaluated by using metrics based on their performance over the set of all bin packing problems, such as the worst case or average case performance. However, this method of assessing a heuristic would only be relevant to a user who employs the heuristic over a set of problems which is actually representative of the set of all possible bin packing problems. On the other hand, a real world user will often only deal with packing problems that are representative of a particular sub-set. Their piece sizes will all belong to a particular distribution. The contribution of this paper is to show that a Genetic Programming system can automate the process of heuristic generation and produce heuristics that are human-competitive over a range of sets of problems, or which excel on a particular sub-set. We also show that the choice of training instances is vital in the area of automatic heuristic generation, due to the trade-off between the performance and generality of the heuristics generated and their applicability to new problems.