Information Theory

Arnon, I., & Ramscar, M. (2012). Granularity and the acquisition of grammatical gender: How order-of-acquisition affects what gets learned, Cognition 122, 292-305.

by Inbal Arnon

Why do adult language learners typically fail to acquire second languages with native proficiency? Does prior... more Why do adult language learners typically fail to acquire second languages with native proficiency? Does prior linguistic experience influence the size of the “units” adults attend to in learning, and if so, how does this influence what gets learned? Here, we examine these questions in relation to grammatical gender, which adult learners almost invariably struggle to master. We present a model of learning that predicts that exposure to smaller units (such as nouns) before exposure to larger linguistic units (such as sentences) can critically impair learning about predictive relations between units: such as that between a noun and its article. This prediction is then confirmed by a study of adult participants learning grammatical gender in an artificial language. Adults learned both nouns and their articles better when they were first heard nouns used in context with their articles prior to hearing the nouns individually, compared with learners who first heard the nouns in isolation, prior to hearing them used in context. In the light of these results, we discuss the role gender appears to play in language, the importance of meaning in artificial grammar learning, and the implications of this work for the structure of L2-training.

Metabolic cost as an organizing principle for cooperative learning

by David Balduzzi

This paper investigates how a population of neuron-like agents can use metabolic cost to communicate the importance of... more This paper investigates how a population of neuron-like agents can use metabolic cost to communicate the importance of their actions. Although decision-making by individual agents has been extensively studied, questions regarding how agents should behave to cooperate effectively remain largely unaddressed. Under assumptions that capture a few basic features of cortical neurons, we show that constraining reward maximization by metabolic cost aligns the information content of actions with their expected reward. Thus, metabolic cost provides a mechanism whereby agents encode expected reward into their outputs. Further, aside from reducing energy expenditures, imposing a tight metabolic constraint also increases the accuracy of empirical estimates of rewards, increasing the robustness of distributed learning. Finally, we present two implementations of metabolically constrained learning that confirm our theoretical finding. These results suggest that metabolic cost may be an organizing principle underlying the neural code, and may also provide a useful guide to the design and analysis of other cooperating populations.

Quantifying causal influences

by David Balduzzi

Common methods of causal inference generate directed acyclic graphs (DAGs) that formalize causal relations between n... more Common methods of causal inference generate directed acyclic graphs (DAGs) that formalize causal relations between n variables. Given the joint distribution of all these variables, the DAG contains all information about how intervening on one variable would change the distribution of the other n-1 variables. It remains, however, a non-trivial question how to quantify the causal influence of one variable on another one.
Here we propose a measure for causal strength that refers to direct effects and measure the "strength of an arrow" or a set of arrows. It is based on a hypothetical intervention that modifies the joint distribution by cutting the corresponding edge. The causal strength is then the relative entropy distance between the old and the new distribution.

We discuss other measures of causal strength like the average causal effect, transfer entropy and information flow and describe their limitations. We argue that our measure is also more appropriate for time series than the known ones.

Finally, we discuss conceptual problems in defining the strength of indirect effects.

Some Qualitative Inferences on Stochastic Matrices via Information Theory

by Emmanuel Gonzalez

Published in the Proceedings of the 5th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment and Management (HNICEM), The Institute of Electrical and Electronics Engineers Inc. (IEEE) - Philippines Section, March 10-13, 2011 at the Traders Hotel in Manila, Philippines.
Co-authored with Aliento V. Estalilla (Ret. De La Salle University Manila) and Tirso A. Ronquillo (Batangas State Uniersity)

Kinds of information in scientific use

by John Collier

Collier, John. Kinds of information in scientific use. Cognition, Communication, Co-operation. Vol. 9 no 2: 295-304.

Abstract: There are many different mathematical definitions of information that have their various uses, but I will be... more Abstract: There are many different mathematical definitions of information that have their various uses, but I will be concerned with notions of information used in applications in various branches of science that are distinguished by their topic, i.e., what they apply to. I describe the major uses information, and show their relations to each other. I will argue that the various uses form a nested hierarchy, in which each is a restriction on the previous, inheriting the properties of its predecessor, but adding in new features that make it a special case. The lowest level is physical information determined by distinctions and the highest is explicit representation in linguistic social communication. Is there anything common to information at all these levels? I will argue that there is, and that information in each case is what Donald MacKay (1969) called a distinction that makes a difference. What distinguishes the use of information at each level is what distinctions make a causal difference at that level. At each successive level distinctions that make a difference at a previous level make no difference at that level. In order to create this sort of filter new levels have to be formed by cohesion peculiar to the identifying characteristics at that level. A consequence of this view is that information must have causal powers, and that there is a tight connection between information and causation.

participative leadership

by Richard Wayman

(2012e) Biological Systems as Systems of Increasing Entropy

by Diego Gabriel Krivochen

DRATF. NEW IMPROVED VERSION COMING SOON. Part of the project "Differentiated linguistic and genetic explorations of language impairments and disorders within a biolinguistic approach on language faculty and language design (UG)", directed by Prof. Dr. Habil. Peter Kosta (Universität Potsdam)

In this paper we will try to provide a formalization of some characteristics of the development of biological systems... more In this paper we will try to provide a formalization of some characteristics of the development of biological systems aiming at the highest level of adequacy: justificative adequacy, that is, not only what we observe (descriptive adequacy) and how we think it occurs (explanatory adequacy) but also what architectural properties (in terms of matter configuration, for example) license the occurrence of certain phenomena. Following the line of Kosta, Peters and Krivochen (2011) –and references therein-, in the case of a biological instantiation of a physical system, justificative adequacy is to be found in the genotype-phenotype dynamics. Moreover, we will claim that this dynamic should be analyzed from a mathematical point of view, since the genotype is hypersensitive to initial conditions: a small change in a certain state of the system may have drastic effects on the output (what is commonly known as “the butterfly effect”). Our inquiry will lead us to revisit the concept of specific language impairment and its relation to other potential phenotypic states of the relevant biological system. Regarding methodology, we will use chaos theory applied to biology and also physical genetics in order to frame our research and give it theoretical weight.

Information retrieval (IR) and the paradox of change: an analysis using the philosophy of Parmenides.

by Clare Thornley

Journal of Documentation, Vol. 68 Iss: 3, pp.402 - 422

“Spells Out The Word of Itself, and Then Dispelling Itself”: The Chaotics of Memory and The Ghost of the Novel in Jeff Noon’s Falling out of Cars

by Andrew Wenaus

Forthcoming: Journal of the Fantastic in the Arts

This article is a study of British author Jeff Noon’s most recent novel Falling out of Cars (2002) as a literary... more This article is a study of British author Jeff Noon’s most recent novel Falling out of Cars (2002) as a literary experiment engaged in raising the ghost of the modern novel, long hailed as dead.  Here, Noon samples canonic literature then transforms, manipulates, and reconfigures it in much the same way a message is transformed when being passed through a communication circuit. The result is a kind of poetic prose Noon terms “metamorphiction”: an elegant experimental mode of fantasy in which signs mutate within certain systemic parameters.  In metamorphiction, the textual past literally haunts the textual present.  This formal experiment is mirrored in the content: the novel concerns a middle aged woman mourning the death of her daughter.  Ultimately, Falling out of Cars is both a virtuosic piece of fantastic fiction and a serious meditation on the contemporary state of the novel.     

Why theories of causality need production: an information-transmission account

by Phyllis Illari

Philosophy and Technology 2011, 24(2) 95-114
DOI: 10.1007/s13347-010-0006-3

In this paper, I examine the comparatively neglected intuition of production regarding causality. I begin by examining... more In this paper, I examine the comparatively neglected intuition of production regarding causality. I begin by examining the weaknesses of current production accounts of causality. I then distinguish between giving a good production account of causality, and a good account of production. I argue that an account of production is needed to make sense of vital practices in causal inference. Finally, I offer an information-transmission account of production based on John Collier's work, that solves the primary weaknesses of current production accounts: applicability and absences.

Functional Complexity: The source of value in biodiversity.

by Keith Farnsworth

doi :10.1016/j.ecocom.2012.02.001
Co Authored with Olga Lyashevska and Tak Fung, who provided a numerical simulation to illustrate this new way of viewing biodiversity. This is a land-mark paper which will shortly appear in the Journal Ecological Complexity.

Biodiversity is not a commodity, nor a service (ecosystem or otherwise), it is a scientific measure of the complexity... more Biodiversity is not a commodity, nor a service (ecosystem or otherwise), it is a scientific measure of the complexity of a biological system. Rather than directly valuing biodiversity, economists have tended to value its services, more often the services of 'key' species. This is understandable given the confusion of definitions and measures of biodiversity, but weakly justified if biodiversity is not substitutable. We provide a quantitative and comprehensive definition of biodiversity and propose a framework for examining its substitutability as the first step towards valuation. We define biodiversity as a measure of semiotic information. It is equated with biocomplexity and measured by Algorithmic Information Content (AIC). We argue that the potentially valuable component of this is functional information content (FIC) which determines biological fitness and supports ecosystem services. Inspired by recent extensions to the Noah's Ark problem, we show how FIC/AIC can be calculated to measure the degree of substitutability within an ecological community. From this, we derive a way to rank whole communities by Indirect Use Value, through quantifying the relation between system complexity and production rate of ecosystem services. Understanding biodiversity as information evidently serves as a practical interface between economics and ecological science.

Conditional Likelihood Maximisation: A Unifying Framework for Information Theoretic Feature Selection

by Adam Pocock

Authors: G Brown, A Pocock, M-J Zhao, M Lujan. Published in JMLR, 2012.

We present a unifying framework for information theoretic feature selection, bringing almost two decades of research... more We present a unifying framework for information theoretic feature selection, bringing almost two decades of research on heuristic filter criteria under a single theoretical interpretation. This is in response to the question: "what are the implicit statistical assumptions of feature selection criteria based on mutual information?". To answer this, we adopt a different strategy than is usual in the feature selection literature−instead of trying to define a criterion, we derive one, directly from a clearly specified objective function: the conditional likelihood of the training labels. While many hand-designed heuristic criteria try to optimize a definition of feature 'relevancy' and 'redundancy', our approach leads to a probabilistic framework which naturally incorporates these concepts. As a result we can unify the numerous criteria published over the last two decades, and show them to be low-order approximations to the exact (but intractable) optimisation problem. The primary contribution is to show that common heuristics for information based feature selection (including Markov Blanket algorithms as a special case) are approximate iterative maximisers of the conditional likelihood. A large empirical study provides strong evidence to favour certain classes of criteria, in particular those that balance the relative size of the relevancy/redundancy terms. Overall we conclude that the JMI criterion (Yang and Moody, 1999; Meyer et al., 2008) provides the best tradeoff in terms of accuracy, stability, and flexibility with small data samples.

Sentence comprehension as mental simulation: an information-theoretic perspective

by Stefan Frank

Frank, S.L. & Vigliocco, G. (2011). Information Sentence comprehension as mental simulation: an information-theoretic perspective. Information, 2, 672-696

It has been argued that the mental representation resulting from sentence comprehension is not (just) an abstract... more It has been argued that the mental representation resulting from sentence comprehension is not (just) an abstract symbolic structure but a `mental simulation' of the state-of-affairs described by the sentence. We present a particular formalization of this theory and show how it gives rise to quantifications of the amount of syntactic and semantic information conveyed by each word in a sentence. These information measures predict simulated word-processing times in a dynamic connectionist model of sentence comprehension as mental simulation. A quantitatively similar relation between information content and reading time is known to be present in human reading-time data.

BLISS: an artificial language for learnability studies

by Sahar Pirmoradian

Across-Language Perspective on Speech Information Rate

by François Pellegrino

On a Non-monotonicity Effect of Similarity Measures

by Jean-Luc Bouchot

Co-authored with Bernhard Moser and Gernot Stübl

The effect of non-monotonicity of similarity measures is addressed which can be observed when measuring the similarity... more The effect of non-monotonicity of similarity measures is addressed which can be observed when measuring the similarity between patterns with increasing displacement. This effect becomes the more apparent the less smooth the pattern is. It is proven that commonly used similarity measures like f-divergence measures or kernel functions show this non-monotonicity effect which results from neglecting any ordering in the underlying construction principles. As an alternative approach Weyl’s discrepancy measure is examined by which this non-monotonicity effect can be avoided even for patterns with high-frequency or chaotic characteristics. The impact of the non-monotonicity effect to applications is discussed by means of examples from the field of stereo matching, texture analysis and tracking.

Exploring Research Data Interactively. Theme One : A Program of Inquiry

by Jon Awbrey

Awbrey, J.L., and Awbrey, S.M. (August 1990), “Exploring Research Data Interactively. Theme One : A Program of Inquiry”, Proceedings of the Sixth Annual Conference on Applications of Artificial Intelligence and CD-ROM in Education and Training, Society for Applied Learning Technology, Washington, DC, pp. 9–15.

If computer programs were smarter, they would, like people, recognize sequences of events, form models of their... more If computer programs were smarter, they would, like people, recognize sequences of events, form models of their environment, and formulate rules based on experience.  This paper describes the development of a program designed to address the difficult computational problems involved in integrating the inductive and deductive reasoning necessary to perform such tasks.  “Theme One” is a prototype program composed of “Index”, a learning algorithm for sequential data, and “Study”, an algorithm for building logical models.  The project goal is an interactive research tool that assists students and investigators in the exploration of qualitative data using artificial intelligence.

An information-theoretic approach to statistical dependence: Copula information

by Rafael Calsaverini

Co-authored with Renato Vicente. Available at arXiv: http://arxiv.org/abs/0911.4207

We discuss the connection between information and copula theories by showing that a copula can be employed to... more We discuss the connection between information and copula theories by showing that a copula can be employed to decompose the information content of a multivariate distribution into marginal and dependence components, with the latter quantified by the mutual information. We define the information excess as a measure of deviation from a maximum-entropy distribution. The idea of marginal invariant dependence measures is also discussed and used to show that empirical linear correlation underestimates the amplitude of the actual correlation in the case of non-Gaussian marginals. The mutual information is shown to provide an upper bound for the asymptotic empirical log-likelihood of a copula. An analytical expression for the information excess of T-copulas is provided, allowing for simple model identification within this family. We illustrate the framework in a financial data set.

"You are cordially invited to a / CHEMICAL WEDDING": Metamorphiction and Experimentation in Jeff Noon's Cobralingus

by Andrew Wenaus

Electronic Book Review
January 2012

This paper is a study of British author Jeff Noon’s writing game, the Cobralingus Engine, from his experimental novel,... more This paper is a study of British author Jeff Noon’s writing game, the Cobralingus Engine, from his experimental novel, Cobralingus (2001).  The aesthetic process of this game whimsically engages with information theory and the remix techniques of electronic music.  In the game, a sample of canonic literature is imaginatively sent through a communication circuit, transformed, manipulated, and finally reconfigured.  The result is a kind poetic prose Noon terms metamorphiction: an amusing and elegant experimental mode in which signs metamorphose and mutate within certain systemic parameters; in short, it is a narrative, not of deconstruction, but of reconstruction. 

Nonlinear Relation Mining for Maintenance Prediction

by Ahmed Mosallam