ART IN THE CONTEXT OF ALGORITHMIC LOGIC PROCEDURES

INTRODUCTION

The artist Neil Harbisson was born with a rare visual syndrome named achromatopsia, that allows him to see only in shades of gray. Since 2004, he has a device called eyeborg implanted in his head that allows him to hear colors. This device works as a receiver with a camera attached at its extremity and it analyzes video flows and converts their colors into sounds. These sounds are transmitted from a chip behind his ear directly into his inner ear. In the beginning, he says to have almost gone mad from the deluge of sounds, but currently he claims to be able to distinguish colors, and comments on how this has changed his behavior and actions in the world (Harbisson, 2012Harbisson, Neil (2012). I listen to color. TEDGlobal 2012. Available in https://www.ted.com/talks/neil_harbisson_i_listen_to_color#t-3259 (Accessed on January 9, 2021).). To provoke changes in one’s own habits and behaviors, as radically as this artist does, is brave and surprising.

Surprising for several reasons. Firstly, by showing how our minds process information, and consequently, how, from this information, we interact with the world. We can assume, based on this case, that there is something outside (of ourselves) and the way that something impresses our senses (natural and artificial) generates an idea, sensation or feeling, and we then create a link between this something from the external world and what was formed internally. Evidently, we are also part of what we call external world, and, for that reason, we feed our formed ideas-sensations-feelings back into the entire system. Also, this established connection must remain logically equivalent, which means that there is a sort of conditional matching. Being conditioned means having a restriction based on criteria that can be of different types: they can be due to similitude, similarity, contiguity, or even a habit, a rule, law, or arbitrary convention. This has been widely studied in sign theories. But this is not the only key for our perceptual results to be functional and to allow interaction with the world efficiently. The brain also seeks to correct any inaccuracies in our senses, and two of its main strategies are «[…] to combine redundant sensory estimates and to use prior knowledge. There is behavioral evidence that the human nervous system employs both of these strategies to reduce the adverse effects of noise and thus to improve perceptual estimates» (Ernst and Di Luca, 2011: 224Ernst, Marc O. and Di Luca, Massimiliano (2011). Multisensory Perception: From Integration to Remapping. In Julia Trommershäuser, Konrad Körding, and Michael S. Landy (eds.). Sensory Cue Integration. Oxford: Oxford University Press, pp. 224-250. DOI: 10.1093/acprof:oso/9780195387247.003.0012. Available in https://www.researchgate.net/publication/215905137_Multisensory_Perception_From_Integration_to_Remapping (Accessed on January 9, 2021).).

The fact is that, in Harbisson’s mind, a connection has been created between the sound and the color of something external, this color having been captured, analyzed, and translated by devices, and this relationship becomes a syntactic instruction for the conceptual system to function and allows it new meanings and forms of interaction with the world. We say syntactic instruction because, even though it is something to be put in the place of something else, without being the represented thing, it also possesses qualities such as frequency, tone, height, duration, rhythm etc., which were associated with other models of representation (sound in particular), and these apparent aspects also gain a dimension of significance in themselves.

Secondly, we have the fact that, after some time, Harbisson recognized that the device had become a part of his body, like an extension of his senses. He reports it took much learning to be able to process sounds and its respective colors effortlessly, almost automatically and immediately. This learning took place through intense repetition, routine, and practice. In other words, the relationship between sound and color was learned. Furthermore, he was able to broaden his senses even more by adding infrared information to the color spectrum that was translated to sound, allowing him to hear colors that the human eye can’t perceive.

Thirdly, in this case we can assume there is a sensory substitution, in which visual data are supplemented by a different channel: hearing; we also have a sensory enhancement, for there was a sensory addition from the infrared data in the transmission (Eagleman, 2020: 91Eagleman, David (2020). LiveWires: The Inside of the ever-changing Brain. Pantheon, Illustrated edition.).

What we can already consider is that: 1. both the sensory and cognitive systems are used upon the creation of relationships between internal and external processes; 2. the brain adjusts to new demands, and this can happen through learning; 3. there is a feedback between internal and external systems, and it corrects our senses’ imprecisions, through the combination of redundant sensory estimates and previous knowledge; 4. from the moment we impose new conditions and experiences, the brain will conform to them.

We are interested in investigating different forms of representation (see the Research Group Realidades at http://www2.eca.usp.br/realidades/en), and these are important contributions to discuss language, technology, thought and art. In this article, we will consider Art an instrument that broadens and questions an aesthetic potential. The artist is the one who explores this field of possibilities. It is intended to demonstrate through art how collaborative processes between mind and algorithms can result in creative actions, and that the artist has an important critical role to play.

1. NEUROPLASTICITY AND INTEGRATION OF SENSES

Though this is a speculative study, there is much evidence that has already been scientifically proven, and the processes of perception and cognition have been the object of many scientific studies. We shall present assumptions from the cognitive sciences that support some of our proposals.

Bryan Kolb, in his book about the plasticity of the brain and behavior, highlights an important issue. He writes that:

That does not mean that everything must be infinitely relearned lest it become lost. On the contrary, the brain also has the capacity to preserve changes, as Kolb reports, using a case study about a woman named only Donna who, even after a ten-year break, had retained great part of her ability to dance, without practicing. This preserved ability allowed her to quickly relearn what she had learned in the past (Kolb, 1995, position 211Kolb, Bryan (1995). Brain Plasticity and Behavior. New Jersey: Lawrence Erlbaum Associates.).

Research shows that the brain is always in activity and should some of its functions diminish or stop (as in the case of someone who, for some reason, goes blind), neighboring territories tend to take control of that region, which is defined as a characteristic of neural plasticity. Another important book is LiveWires: The Inside of the ever-changing Brain, by David Eagleman, that explains the previously mentioned case of Neil Harbisson (Eagleman, 2020: 84Eagleman, David (2020). LiveWires: The Inside of the ever-changing Brain. Pantheon, Illustrated edition.), aside from providing a historical overview of neuroscience studies and other instigating examples that contribute to the subject matter.

Right at the beginning of his book, Eagleman states that our brain machinery is not entirely preprogrammed, it molds itself as it interacts with the world throughout our lifetime. This does not seem that new, as can be illustrated with some references from other fields: a) John Dewey (educator) has shown that the way we think reflects the experience we have had (Dewey, 2010Dewey, John (2010). Arte como Experiência. São Paulo: Ed. Martins Fontes-Selo Martins.); b) the concept of Umwelt, explored by Jacob von Uexküll (biologist) since 1909, works from the premise that life is an open and coherent system where subject and object are defined as interrelated elements in a larger context (Von Uexküll, 2007Von Uexküll, Thure (2007). A teoria da Umwelt de Jakob von Uexküll. Galáxia. Revista do Programa de Pós-Graduação em Comunicação e Semiótica, 7: 19-48. Recuperado de https://revistas.pucsp.br/index.php/galaxia/article/view/1369 (Accessed on January 9, 2021).); c) Furthermore, an approach to the ideas of individuation and relational transduction of the philosopher Gilbert Simondon (2009) Simondon, Gilbert (2009). On Gilbert Simondon. Parrhesia, 7 (2-3). Available in http://www.parrhesiajournal.org/parrhesia07/parrhesia07.pdf (Accessed on January 9, 2021). would also bring important developments for this investigation.

Nonetheless, the innovator in Eagleman is, mainly due to the number of registered cases, the pragmatic way he arguments, employing scientifically proven case studies and occurrences, demonstrated by brain imaging from various techniques; in addition to presenting findings in synesthesia and neurotechnological devices to try to answer the question, «How can a blind person learn to see with her tongue, or a deaf person learn to hear with his skin? » (Eagleman, 2020: 4Eagleman, David (2020). LiveWires: The Inside of the ever-changing Brain. Pantheon, Illustrated edition.). Despite being difficult to verify, he systematically shows that, in general terms, the brain is configured with basic building blocks and will mold itself with what it absorbs from its surroundings. The point here is that the brain

This is an important point for us, given we assume that our mind molds itself from the representational models of each culture, and we are currently overwhelmed by what we call conformed thoughts, which we will see subsequently, and this must be causing substantial changes in our way of thinking and acting (Laurentiz 2017Laurentiz, Silvia (2017). Videogames e o desenvolvimento de habilidades cognitivas. DAT Journal, 2 (1): 80-90. https://doi.org/10.29147/2526-1789.DAT.2017v2i1p79-89. Available in https://datjournal.anhembi.br/dat/article/view/45/37 (Accessed on January 9, 2021)., 2018Laurentiz, Silvia (2018). Conformed Thought: Consolidating Traces of Memories. In Aaron Marcus and Wentao Wang (eds.). Design, User Experience, and Usability: Users, Contexts and Case Studies. Cham: Springer. pp. 28-40., 2019Laurentiz, Silvia (2019). Conformed thought and the Art of Algorithms. In Priscila Arantes, Vítor J. Sá, Pedro Alves da Veiga, and Adérito Fernandes-Marcos (eds.). Proceedings of the 9th International conference on digital and interactive arts - ARTECH. Portugal: Editora Universidade Católica Portuguesa,pp. 465-472. ISBN/ISSN 9781450372503).

Neuroplasticity is the term used by neuroscience for this ability of the brain to change from experience and retain certain changes. But, as the word plastic is loaded with other meanings and temporality, the author considers it appropriate to use another term: livewired (Eagleman, 2020:14, position 273Eagleman, David (2020). LiveWires: The Inside of the ever-changing Brain. Pantheon, Illustrated edition.). Thus, it defines the brain as a dynamic, adaptable system in search of interconnected and networked information¹ »[...] the days of being impressed by plastic molding may be past us. Our goal here is to understand how this living system operates, and for that I’ll coin a term that better captures the point: “livewired”. As we’ll see, it becomes impossible to think about the brain as divisible into layers of hardware and software. Instead, we’ll need the concept of liveware to grasp this dynamic, adaptable, information-seeking system «(Eagleman, 2020: 14)..

The number of scientific articles on brain plasticity is extensive nowadays, but it suffices at this moment to highlight this characteristic of self-configuration and adaptability, and to say that neural territories can reconnect and assume functions that are missing or damaged, and still to understand that what we are, and think, is the result of interaction with our surroundings, based on the experiences we have had. This also means that brain network processes involve more than genetics, and that the brain can change as a direct result of experience.

Add to this the fact that, «[…] in the past decade, however, several independent lines of research have demonstrated cross-modal responses in primary sensory areas […]» and that the results of these surveys «support the notion that primary sensory areas have a preference for a given modality but can engage in meaningful cross-modal processing depending on task demand» (Vasconcelos et al., 2011: 15408Vasconcelos, Nivaldo; Pantoja, Janaina; Belchior, Hindiael; Viegas Caixeta, Fábio; Faber, Jean; M. Freire, Marco Aurelio; Cota, Vinícius; de Macedo, Edson Anibal; Laplagne, Diego A.; Martins Gomes, Herman; and Ribeiro, Sidarta (2011). Cross-modal responses in the primary visual cortex encode complex objects and correlate with tactile discrimination. Proceedings of the National Academy of Sciences, 108 (37): 15408-15413. DOI: 10.1073/pnas.1102780108. Available in https://www.pnas.org/content/108/37/15408 (Accessed on January 9, 2021).).

This means that Science has reviewed the idea that each primary cortical area would represent a highly specialized functional target (one area that would be intended for sight, another for smell, hearing, etc.) and that primary sensory areas would be dedicated exclusively to the processing of a single modality sensory, while multisensory integration would occur only in higher-level associative areas. Recent research indicates that the primary sensory areas prefer a dominant modality, but they are also capable of cross-modal processing (Vasconcelos et al., 2011Vasconcelos, Nivaldo; Pantoja, Janaina; Belchior, Hindiael; Viegas Caixeta, Fábio; Faber, Jean; M. Freire, Marco Aurelio; Cota, Vinícius; de Macedo, Edson Anibal; Laplagne, Diego A.; Martins Gomes, Herman; and Ribeiro, Sidarta (2011). Cross-modal responses in the primary visual cortex encode complex objects and correlate with tactile discrimination. Proceedings of the National Academy of Sciences, 108 (37): 15408-15413. DOI: 10.1073/pnas.1102780108. Available in https://www.pnas.org/content/108/37/15408 (Accessed on January 9, 2021).).

Another interesting fact, returning to Eagleman’s book, is how the brain needs to practice repeatedly to learn an activity, which can be motor or cognitive. This practice significantly changes brain configuration, so much so that «[…] when medical students study for their final exams over the course of three months, the gray matter volume in their brains changes so much it can be seen on brain scans with the naked eye» (Eagleman, 2020: 143, position 2308Eagleman, David (2020). LiveWires: The Inside of the ever-changing Brain. Pantheon, Illustrated edition.).

That leads us to recover the studies of Gestalt theory (1935), which discovered already at that time that the performance of a task depends on previous performances, and the concept of memory traces is an attempt to explain this dependency. It stated that it was not easy to distinguish an innate process from an acquired one, but the interesting point is that «an experienced tennis player did not learn to perform a small number of specific movements, but to hit the ball correctly in the multivariate situations of the game» (Koffka, 1975: 516Koffka, Kurt (1975). Princípios de Psicologia da Gestalt. São Paulo: Cultrix.). This means that, in the learning process, we create systems of traces of specific types, consolidate them and make them increasingly accessible, whether in repeated or new situations. Hence, learning is defined by traces of memories that are learned, consolidated and available, and that modify processes and, consequently, behaviors. Gestalt practitioners also considered that traces could be transformed through interaction with other traces and processes. Therefore, surrounding experiences are not only about the memorizing of objects or connections between things of the world and some internal relationship created with these things, but also the exercising of cognitive skills, making them accessible to new functions through learning processes. Thus, we reiterate, from the point of view of neuroscience and psychology, the points raised in the introduction of this article, relating to the experience of artist Neil Harbisson. We now recognize, in addition to those mentioned previously, the brain’s capabilities for self-configuration, adaptability and multisensory integration.

We have already pointed out in previous articles the relationship between sensory experiences and the representational aspects of experiences (Laurentiz, 2017Laurentiz, Silvia (2017). Videogames e o desenvolvimento de habilidades cognitivas. DAT Journal, 2 (1): 80-90. https://doi.org/10.29147/2526-1789.DAT.2017v2i1p79-89. Available in https://datjournal.anhembi.br/dat/article/view/45/37 (Accessed on January 9, 2021)., 2018Laurentiz, Silvia (2018). Conformed Thought: Consolidating Traces of Memories. In Aaron Marcus and Wentao Wang (eds.). Design, User Experience, and Usability: Users, Contexts and Case Studies. Cham: Springer. pp. 28-40., 2019Laurentiz, Silvia (2019). Conformed thought and the Art of Algorithms. In Priscila Arantes, Vítor J. Sá, Pedro Alves da Veiga, and Adérito Fernandes-Marcos (eds.). Proceedings of the 9th International conference on digital and interactive arts - ARTECH. Portugal: Editora Universidade Católica Portuguesa,pp. 465-472. ISBN/ISSN 9781450372503), which culminated in what was called «Conformed Thought».

Conformed Thought is a sign, although not every sign is a conformed thought. To understand this statement, we must recognize the different types of signs that form the thought. According to Charles Sanders Peirce:

Therefore, the Conformed Thought would be in the third category of symbols, the one related to the habit(s) acquired. Thus, Conformed Thought is cultural and context-dependent, and is related to a technology of its time.

2. CONFORMED THOUGHT: PREVIOUS STUDIES

Conformed Thoughts are patterns, codes and set of codes, which end up giving rise to new representational models, always considering this intrinsic condition of the relationship between experience, sensations, and cognition (Laurentiz, 2015Laurentiz, Silvia (2015). Sensoriality and Conformed Thought. In: Constantine Stephanidis and Margherita Antona (eds.). Universal Access in Human-Computer Interaction. Access to Interaction. UAHCI 2015. Lecture Notes in Computer Science, vol 9176. Springer, Cham. Pp. 217-225. https://doi.org/10.1007/978-3-319-20681-3_20 ).

Among Peirce’s principles, there are different types of signs that make up our thoughts, and there is one that is a vague composition of sensations, emotions, and feelings. This type is governed by our sensory system (including our sense of experience and observation), and it also leads our thoughts towards semiotic developments, promoting changes in habits and giving rise to new signs. Amidst the tension between sensitive and conforming thoughts, there is a fertile environment of conflicts, interferences, correspondences, tensions, scenarios, and a mixture of information (Laurentiz, 2015Laurentiz, Silvia (2015). Sensoriality and Conformed Thought. In: Constantine Stephanidis and Margherita Antona (eds.). Universal Access in Human-Computer Interaction. Access to Interaction. UAHCI 2015. Lecture Notes in Computer Science, vol 9176. Springer, Cham. Pp. 217-225. https://doi.org/10.1007/978-3-319-20681-3_20 ).

The Colors and Words Stroop test (developed by John Ridley Stroop) can be a didactic example to explain what we mean. It was developed to evaluate the automation of the reading process, in addition to assessing selective attention, focus and speed in processing information. The task is to name the color in which each printed word is written (which in turn are words named after colors), where there is a divergence between the written word and the color of ink that was used to write the word (the word green is written in yellow ink, for example). In general, written words strongly interfere with our ability to identify and name the colors of those words. Thus, the interference generated in the processing of divergent information (the word itself and the word’s color) by the brain creates a conflicting message. Neuroscience tries to explain the reasons for this phenomenon, but for the moment the important thing is to focus on the effort needed to perform this simple task, which demonstrates how we are guided by habits (the learned words and their meanings), although we are not always aware of it. This implies that all abstract thinking must create an interference of some kind in the way we perceive and act in the world (Laurentiz, 2015Laurentiz, Silvia (2015). Sensoriality and Conformed Thought. In: Constantine Stephanidis and Margherita Antona (eds.). Universal Access in Human-Computer Interaction. Access to Interaction. UAHCI 2015. Lecture Notes in Computer Science, vol 9176. Springer, Cham. Pp. 217-225. https://doi.org/10.1007/978-3-319-20681-3_20 ). In this way, the Colors and Words Stroop test and the commented case of artist Neil Harbisson help us to realize an important point: comparatively, we can consider that patterns, codes, and code sets (conformed thoughts), especially of computer languages, will also be offering a similar experience, where their sensitive qualities of updates and interfaces (and even those that we do not even notice their presence) act on us and cause changes in our behavior² To advance in this subject see Silvia Laurentiz (2017, 2018 and 2019)..

In my article on Video Games and the development of cognitive skills, the proposal was to distinguish a mechanical-visual cognitive development of videogame players, originating from an initial report by a player who said that some games were able to train and improve performance in other games, and who claims to have benefited from that (Laurentiz, 2017Laurentiz, Silvia (2017). Videogames e o desenvolvimento de habilidades cognitivas. DAT Journal, 2 (1): 80-90. https://doi.org/10.29147/2526-1789.DAT.2017v2i1p79-89. Available in https://datjournal.anhembi.br/dat/article/view/45/37 (Accessed on January 9, 2021).). In other words, we were proposing that it is possible to exercise cognitive skills, which are associated with gameplay, and which are required in most games. This was a speculative hypothesis, especially because, as already pointed out by Gestalt theorists, it is not simple to distinguish an innate process from an acquired one, but as an initial data it relied on that empirical report.

Gestalt served as a basis at that time, proposing that we are able to train skills that may be requested in different situations, and not just those of the training itself. Therefore, once we define learning as memory traces that were learned, consolidated, and made available, we arrived at some propositions at that point:

In the article Conformed Thought: Consolidating Traces of Memories, we indicated at that moment that there is an important passage when, from the image of something, we move on to the mathematical model of that thing. The modelling process itself already contributes to what we were studying. Right from the start, we highlighted the distance between thing in itself in the world, thing in the world that has been objectified (made object), and model of an objectified thing. From a different angle, models are formed by objects, which in turn are objectified things (Laurentiz, 2018Laurentiz, Silvia (2018). Conformed Thought: Consolidating Traces of Memories. In Aaron Marcus and Wentao Wang (eds.). Design, User Experience, and Usability: Users, Contexts and Case Studies. Cham: Springer. pp. 28-40.).

It is evident that there is a tension between thing in itself and object (indeed, as we have said, all abstract thinking creates an interference of some kind in the way we perceive and act in the world), which sparks significant gains and losses through the sign process. This tension already establishes a degree of abstraction. However, the transition from an object to a model, which is when it becomes what we call Conformed Thought, carries a new degree of abstraction, and that was what truly set off the research.

It should be made clear that the very definition of object is not simple. The concept used at that time was that object is thing that was objectified, in which objectifying means giving expression, whether to an abstract notion, a feeling, an idea, or anything, in a way that can be experienced by others. That is, it provides conditions for things to be experienced and shared. This is important because, as previously demonstrated, our mind is constantly changing, self-configuring, adapting to demands, and in multisensory integration with our surroundings, and our premise is that the way in which experience and the sharing of information are enabled plays a fundamental role in this process.

There is currently an important philosophical movement known as Object-Oriented Ontology (OOO), often associated with the Speculative Realism. One of the founders of this movement is professor Graham Harman. However, our research is based on Charles Sanders Peirce’s theory, as already aforementioned.

In a possible presentation of Peirce’s realism, Ivo Ibri explains that it would not be «[...] um realismo que admite, tão somente, a existência das coisas externas à nossa interioridade, mas, sim, aquele que reconhece a realidade de um tecido de generalidade, conatural, em sua idealidade, ao nosso pensamento» (Ibri, 2020: 36Ibri, Ivo A. (ed.) (2020). Semiotics and Pragmatism. Theoretical Interfaces (Vol. 1). São Paulo: Marília, Editora Cultura Acadêmica.).

Peirce brings a special relationship with the objects that are offered to our perception: the relationship between immediate and dynamic object³ »Namely, we have to distinguish the Immediate Object, which is the Object as the Sign itself represents it, and whose Being is thus dependent upon the Representation of it in the Sign, from the Dynamical Object, which is the Reality which by some means contrives to determine the Sign to its Representation« (Deeley, 1994: Collected Paper 4.536)..

According to Winfried Nöth,

Thus, it is evident to realize that an apple (thing), the image of this apple (objectified thing, therefore object) and the mathematical model of this apple (synthesis of the object), already indicate important representational aspects for brain conformation. The author who deals with these sign passages very well is Vilém Flusser (2002Flusser, Vilém (2002). Filosofia da Caixa Preta. Ensaios para uma futura filosofia da fotografia. Rio de Janeiro: Relume Dumará, coleção Conexões., 2007)Flusser, Vilém (2007). O mundo codificado. São Paulo: Cosac Naify. and, to him, they are indicative of degrees of abstraction. It is crucial to understand that all this happens in a context that feeds back into the system, and that, consequently, there are also processes involved in evaluation, transformation, comparison with reference values, adaptation, and coding. This is how the cybernetic feedback model we are following works (Laurentiz, 2018Laurentiz, Silvia (2018). Conformed Thought: Consolidating Traces of Memories. In Aaron Marcus and Wentao Wang (eds.). Design, User Experience, and Usability: Users, Contexts and Case Studies. Cham: Springer. pp. 28-40.). Thus, if we have already said that, from neuroscience, it is clear that the brain uses strategies to correct inaccuracies in the information received by our senses and that two of them are combining redundant sensory data and prior knowledge, at this time, we are adding new skills. That is, in the process of using objects with the purpose of sharing with others our acquired experiences, we employ several strategies, and with them we cause effects (ideas, sensations and feelings) and conformed thoughts, which feed back into the system -meaning they share with other language systems, objects and models in processes of evaluation, transformation, comparison of values, etc.-, in a continuous and circular relationship. We will address this point in more detail later.

Let’s return to the main topic after that brief digression to clarify a few points. According to Gualtiero Piccinini and Andrea Scarantino (2011)Piccinini, Gualtiero and Scarantino, Andrea (2011). Information processing, computation, and cognition. Journal of Biological Physics 37: 1-38. DOI 10.1007/s10867-010-9195-3., the notions of computing and information processing are often confused and end up being associated with a theory of cognition. The authors warn about the differences between them, making it clear that the notion of information is related to communication systems and bears issues of signs; and that information processing through computation can only be analogously compared to a cognitive system. In this article, information processing must be carried out by means of computation in the generic sense (Piccinini and Scarantino, 2011: 26Piccinini, Gualtiero and Scarantino, Andrea (2011). Information processing, computation, and cognition. Journal of Biological Physics 37: 1-38. DOI 10.1007/s10867-010-9195-3.).

It should be clear that we are not considering information processed at the machine level, where each memory cell finds two states, 0 and 1, nor are we talking about sequences of bytes or bits, as they do not carry semantic content. We are considering Information as a collection or set of organized data and knowledge, including software, systems, and the programming itself that will be carrying out logical procedures, manipulation, and production of images, and may even generate information to some semantic degree in a mathematical and probabilistic way.

Furthermore, using digital-analog and analog-digital converters, the system will carry natural and nonnatural information, preserving some causal relationship between them. Thus, generically, computation and cognition can be compared, but the authors warn that it is not so simple:

Surely, these are legitimate questions but will not be discussed in this article. Anyway, the concept of information has been used in different ways, depending on the context in which it is inserted and under which a specific phenomenon is related. In this article, information is what will be transmitted by the sign, which means that it is part of a chain of significant actions, led by interpretants, which Peirce called semiosis.

Depending on the nature of the sign, as already explained, it will transmit certain information, which will also depend on the context and knowledge of its interpreter. Interestingly, as Peirce is not anthropocentric, «each increase of a man’s information is at the same time the increase of a word’s information and vice versa. So that there is no difference even here» (Deeley, 1994: Collected Paper 7.587Deeley, John (ed.) (1994). The Collected Papers of Charles Sanders Peirce. Electronic Edition. Disponible: https://colorysemiotica.files.wordpress.com/2014/08/peirce-collectedpapers.pdf.).

Back to the matter, in the article Conformed Thought and the Art of Algorithms, we discussed logical procedures in conceptual and perceptual models, in a case study of works belonging to a relationship between art, science and technology (Laurentiz, 2019Laurentiz, Silvia (2019). Conformed thought and the Art of Algorithms. In Priscila Arantes, Vítor J. Sá, Pedro Alves da Veiga, and Adérito Fernandes-Marcos (eds.). Proceedings of the 9th International conference on digital and interactive arts - ARTECH. Portugal: Editora Universidade Católica Portuguesa,pp. 465-472. ISBN/ISSN 9781450372503). Our interest was to reflect on specific contributions of codes and set of codes, algorithms, and patterns, as resulting of concepts, texts, and scientific theories and, therefore, as updated forms of elaborated knowledge (conformed thought). It was a study aimed at understanding how these logical procedures shape thinking, and how they feed into the systems -sensory and cognitive- that evolutionarily and through circularity generate increasingly complex sign systems. We were, at that time, focused on the different logical processes of algorithms because, considering the improvement in the development of algorithms in the last decades, it is remarkable how we can perceive changes in the approaches and processes used by the artists.

Among the different logical processes involved in the algorithms discussed, we mention pseudo-random, stochastic, cybernetic, emergence, generative processes, and lastly, processes of artificial intelligence and learning. We also present the differences between a classic algorithm and a machine learning one.

It is only necessary to consider that, while classic algorithms produce an output based on the steps described in the sequence of instructions of the algorithm, similarly to a cake recipe, in the machine learning algorithm, a mathematical model is built that maps inputs to specific outputs, and then feeds the model with pairs of input + expected output to train it -and it will adjust its internal parameters from this training.

That is, there is a difference in procedures and results, which in some way means that the algorithm itself goes through a process of modeling and adjustments determined by what it learned during its training stage. With the training completed, the program will be able to be used with new data entries -and even be easily adapted to new situations. It would be hasty to suggest that it would thus be consolidating memory traces, but we must recognize that there is a difference between an algorithm that solves a problem and one that can come to solve different problems.

But at that moment we were interested in recognizing that, in the generation of models and between models, simulations and learning, we have different strategies involved. Classification technique, for example, widely used in Machine Learning, groups things that are similar by parameters that satisfy some selection criteria, norm, or law. Discrimination processes, also extensively used, impose restrictions based on certain conditions and circumstances. Therefore, we not only have synthesis of objects transformed into models, but we also have mathematical models that analyze, classify, and select models, which are formed by objects, which are objectified things. Indeed, there are classic internal algorithms within the Machine Learning algorithm, encapsulated algorithms, networks containing several coupled algorithms. This increases the complexity of these logical procedures and, in turn, the levels of representational abstraction. These new procedures should cause a new tension between sensations and conformed thinking, in an environment of blends of information and levels of abstractions.

3. ART AND LOGICAL PROCEDURES OF THE ALGORITHMS

Initially, we must understand that the digital image production processes are different from analog processes. It is, substantially, a formal procedure, that stores elements described in the form of data, and uses abstract rules in its construction, restricting its state space, or field of possibilities, to what can be mathematically well defined. However, despite their formal logical-abstract features, they carry plasticity-sensitive marks -by updating themselves formally. Evidently, for mathematical expressions to be viewed as images, it will take more than simple translation of numerical data into the projected light information. As Arlindo Machado would say, «[...] é preciso, antes de mais nada, posicionar-se em relação a certos ditames da representação plástica acumulados ao longo dos séculos, tal como nos foram legados pela tradição da pintura, da fotografia e do cinema» (Machado, 1993: 60Machado, Arlindo (1993). A Simulação da ImagemIn: Máquina e Imaginário: O desafio das Poéticas Tecnológicas. São Paulo: Edusp, pp. 59-112. ).

This is an important issue because, for the most part, we are just forging procedures so that the image gains characteristics of representations which already have some sort of consented ideological domination. This means that the algorithm will simulate a model of perspective projection, and «[…] esse modelo depende basicamente da maneira como imaginamos a intervenção do observador na enunciação da imagem, como por exemplo: os modelos de perspectiva paralela e a clássica» (Machado, 1993: 65Machado, Arlindo (1993). A Simulação da ImagemIn: Máquina e Imaginário: O desafio das Poéticas Tecnológicas. São Paulo: Edusp, pp. 59-112. ). This is important because we will see later some examples of works that seek to simulate representational models, and it depends on us to realize that, even when viewing a digital image, we resort to earlier projective techniques.

Furthermore, returning to the ideas of Vilém Flusser, when we talk about images produced by apparatus⁴ There are many distinctions between apparatus and device that would be interesting to explore, but for this article both will be considered as producers of thoughts conformed at some level., we already refer to processes of abstraction through two capacities: imagination and conceptualization. Imagination is the ability to encode four-dimensional phenomena into flat symbols and to decode the information thus encoded. In a general sense, imagination is the ability to make and decipher images. Conceptualization, on the other hand, is the ability that allows the encoding and deciphering of texts (Flusser, 2002: 10Flusser, Vilém (2002). Filosofia da Caixa Preta. Ensaios para uma futura filosofia da fotografia. Rio de Janeiro: Relume Dumará, coleção Conexões.).

Hence, a traditional image, for Flusser, is a first-degree abstraction because it abstracts two dimensions from the concrete phenomenon that has four dimensions. In contrast, the technical image, the one produced by an apparatus, is a third-degree abstraction, since apparatus are applied scientific texts (second-degree abstraction) and, therefore, concepts. Consequently, the process of abstraction of the technical image uses both capacities of abstraction, namely, imagination and conceptualization.

Yet there is still, as recognized by Flusser, the emergence of «a new imagination» where «[...] somente uma Imaginação totalmente calculada pode ser considerada explicada», with a character of simulation, «[...] como se a imaginação tivesse se autonomizado» (Flusser, 2007: 169, 162, 173Flusser, Vilém (2007). O mundo codificado. São Paulo: Cosac Naify.).

It is still called imagination by the author because it creates and deciphers images, but it is a different type of image. The images of the new imagination are projected by dimensionless calculations (Flusser, 2007: 173Flusser, Vilém (2007). O mundo codificado. São Paulo: Cosac Naify.).

The intention of a synthesized image may be similar to that of other conventional images, one may want to imitate a circumstance or object, but then the new imagination will have been put at the service of the old (Flusser, 2007: 174Flusser, Vilém (2007). O mundo codificado. São Paulo: Cosac Naify.). Recalling Machado (1993)Machado, Arlindo (1993). A Simulação da ImagemIn: Máquina e Imaginário: O desafio das Poéticas Tecnológicas. São Paulo: Edusp, pp. 59-112. , the digital image visualization process itself already simulates representational models from previous processes. We currently have numerous works using AI for image production that demonstrate that this issue is quite complex. We will see below examples of images generated by an AI algorithm that acts as a style generator. Would they be putting the new imagination at the service of a first-degree imagination that of traditional images, in this case?

The important thing to note for now is that even if you want to imitate/simulate objects that are things of the world from the construction of models, they may be logically equivalent to the simulated object -but not the same. For a closer look, this distinction is important.

Thus, something (with four dimensions) is abstracted into image of this thing (representation with two dimensions through the capacity of imagination), which is abstracted into a concept of the image of the thing (one dimension, linear or discursive writing through the capacity for conceptualization), which is then abstracted into the image of the concept of the image of the thing (product of apparatus, which in turn are applied scientific texts that are reconstructed on a two-dimensional surface), which is then abstracted into synthetic image by simulation/modeling of the object (generation of a model of an objectified thing, or mathematical model of procedures, by dimensionless synthesis), by a new capacity for imagination. In each of these processes there will be gains and losses, which will result in new experiences (exchanges, sharing, interaction, feedback, and so on) and demands, which will con/in/form our brain, and which, consequently, will change our behavior and actions in the world. Ultimately, we act in the world like Neil Harbisson, perhaps not so radically, but it is important to be aware of that.

4. CASE ANALYSIS

How does art contribute to this issue? We have already said that we will consider Art as that knowledge that expands and questions an aesthetic potential, and that the artist is the one who explores this field of possibilities. In this sense, we will present works that we consider bringing an expansion of aesthetic issues related to conformed thinking, since they use AI algorithms (autonomous imagination) in their development. However, of course, these types of works are not limited to what we will present next, which will only serve for our final considerations on how conceptual and perceptual models of current works of art generate aesthetic dismemberments.

Let us return to the case of an image generated by an AI style transfer program, which seeks to simulate a representational model based on the analysis of a database of numerous images of a specific style. We have already cast doubt on whether an autonomous imagination of this type would be at the service of a first-degree imagination, that of traditional images. This question is appropriate because, in a way, the resulting images recover «[…] certos ditames da representação plástica acumulados ao longo dos séculos [...]» (Machado, 1993: 60Machado, Arlindo (1993). A Simulação da ImagemIn: Máquina e Imaginário: O desafio das Poéticas Tecnológicas. São Paulo: Edusp, pp. 59-112. ), to extract a profile that will be used as a model for new images. As in any computer calculation product, the result of such a process, in the search for transferring a style from an image database, there is the extraction of a statistical relationship of some kind and, therefore, a mathematical calculation.

We can cite several works that follow this reasoning.

We are not questioning the artistic merit of these productions, only stressing the importance of realizing that what is admirable in these works is the mathematical calculation capable of generating such a complex model, produced by other models of image production. In addition, they are conformed thoughts, governed by an autonomous imagination, with a character of image simulation, and with dimensionless abstraction (Flusser, 2007Flusser, Vilém (2007). O mundo codificado. São Paulo: Cosac Naify.). Another relevant factor is the amount of data used, that in other processes would not be configurable, and that now generate statistical models (like an analytical intelligence way).

As Ron Augustus explains, in the project The Next Rembrandt⁵ Cooperative work between ING, Microsoft, Delft University of Technology, and two museums: the Mauritshuis and the House of Rembrandt. Available in https://www.nextrembrandt.com/ , developed by Microsoft:

Therefore, although an image has been created by placing the new imagination at the service of the old (Flusser, 2007Flusser, Vilém (2007). O mundo codificado. São Paulo: Cosac Naify.), and the visualization process simulates representational models from previous processes (Machado, 1993Machado, Arlindo (1993). A Simulação da ImagemIn: Máquina e Imaginário: O desafio das Poéticas Tecnológicas. São Paulo: Edusp, pp. 59-112. ), it can be seen that the aesthetic issue is in the conceptual model generated, more than in the plastic result achieved:

Jorge Albuquerque Vieira believes that the artist, exploring fields of possibility in his/her surroundings, «[…] tangencia a realidade e percebe coisas que muitas vezes nem um cientista percebe» (Albuquerque Vieira, 2010: 22Albuquerque Vieira, Jorge (2010). Teoria do Conhecimento e Arte. Revista Música Hodie, 9 (2). Available in https://www.revistas.ufg.br/musica/article/view/11088/7310 (Accessed on January 9, 2021).).

We can mention another work that explores this process as well in 2018. The collections La Famille De Belamy produced by the Parisian collective Obvious-art, has generated many discussions, especially due to the high value that it was commercialized for. Questions about the art market do not engage us at this point. What matters in this article are the logical procedures involved. Currently, neural network architectures are composed of more than one network, which are already quite complex by themselves. Each one is placed as if they were competitors and opponents of each other. The two competing neural networks (GANs) used were called Generator and Discriminator. The first, as the name suggests, creates an image from information fed by a database. The authors used a database of over 10,000 portraits painted between the 14th and 20th centuries. The Discriminator tries to find differences between images made by human beings and those made by the Generator. The idea is for the Generator to deceive the Discriminator, so that the latter considers that the image has been produced by a human and, therefore, would not be synthetic but original, which would lead it to be selected. In this process, the system could learn to distinguish more and more quickly what would be a synthetic image from one produced by human beings. Explicitly, we have a case of an autonomous, dimensionless imagination passing for a first-degree imagination. Again, the biggest contribution in this case, in our view, is the developed mathematical model that is capable of simulating a representational model of the first-degree abstraction process, and this passage in some way leads us to perceive something that we had not noticed before in those particular aesthetic models in a formal analysis of works.

The next piece is completely different. The video work entitled What I saw before the darkness, by an artist who simply calls herself the girl who speaks with AI, was created by artificial intelligence algorithms as well. Initially, a hyper-realistic image of a human face was generated. The program creates a face, a synthetic image generated by algorithms through an autonomized Imagination, as the face created does not exist in real life. Instead, it was composed by neural networks (GANs) that learn not from paintings, as in the previous examples, but from photos of people’s faces, to produce an image from a synthesis model extracted from this database. In this case, a machine was trained from millions of photographs to produce a realistic human face, but that does not represent any person who has ever lived.

Then, one by one, the neurons of the artificial mind are turned off, while the process is recorded in a time-lapse video. It does not matter which and how data are erased, but that with each erasure, the face is transformed into a completely new image. It is a metaphor, in which the network would slowly forget how that face looks. As if the artificial neurons were turning off one by one, simulating a forgetfulness, until the network effectively forgets the appearance of that face and stops recognizing a face in the resulting image.

We have several important issues with this work, but we will highlight only three. Firstly, the idea of a photograph of someone who does not and has not existed is already disturbing. It harkens back to the whole history of photography, from the relation of the indices of reality that a photograph is capable of rescuing; yet, despite the recognition of traces of photography, there does not exist the thing itself photographed. That is, its referential is not a singular referent, but a generalized synthesis of innumerable referents. Secondly, the project creates a metaphor for the functioning of the brain system based on the use of neural networks and AI algorithms. The network creates an image with a face from countless images; there is the recognition of what a face is, the identification of the elements that make up a face, the discrimination of a face, and comparison between countless faces, until the moment when the information loses its reference values and criteria necessary to identify one face. But what the network is unable to perceive is that when we watch the process recorded on video, we start to see in those images’ other forms: a disfigured face, a skull, the representation of death, and we start to create other senses and meanings. In other words, what is surprising to us the system cannot perceive, losing this aesthetic potential. Thus, the system may even metaphorically represent some cognitive abilities, but it is not able to admire its aesthetic potential. Thirdly, it creates a provocation about the role of forgetfulness in our lives. Recalling Funes o Memorioso, by Jorge Luis Borges, forgetting has an important role for thought. «To think is to forget a difference, to generalize, to abstract» (Borges, 1979Borges, Jorge Luis (1979). Funes el memorioso. In Prosa Completa, Barcelona: Ed. Bruguera.). And Milan Kundera would say that in the face of life, «the old dead [should] make room for the young dead» (Kundera, 1985Kundera, Milan (1985). Risíveis Amores - Sete histórias de amor extremamente originais. Brasil: Editora Nova Fronteira.). What is distressing is the paradox of proposing for a computer to forget data.

As a last example we will mention the artist Sougwen Chung. The artist defines her work as human-robotic design performances, in full collective collaboration, exploring a work that emerges from human and non-human relationships. She creates performances with one or more robotic arms, which respond to a variety of data inputs that she has been developing for some time. As explained by the artist, in the beginning she used a gesture-based approach, using a neural network. The drawing of the line that the artist made manually was recorded in real time, either by an aerial camera or by a sensor at the tip of the paintbrush, which transformed her positional data into something that could be read by the system. The system articulated by the robotic units produced a set of positions based on an interpretation of the artist’s own drawing archives over the last twenty years. It is a process that builds a feedback cycle of the artist’s own drawing style with the drawings that are being made in the act of performance, between the artist and the robotic arm, as if it were a graphic dance done by both. The artist recently integrated other data into the system, such as her biometrics, with data on her heart rate and brain waves. The artist idea is to «think about ways in which humans connect to mechanical and artificial systems and vice versa and ways that can function as a creative catalyst». Again, several important issues emerge from this work, we will only highlight some.

First is the very definition of drawing made from the artist’s gesture, and its replication by the robotic arm. The idea that the drawing of the artist’s stroke can be recovered by devices (be cameras or sensors) and that this data would define her drawing seems initially limited, but it expands the possibilities of interaction between the drawing from the artist’s gesture and body and the program with the interface of the robotic arm. It is natural that passing these data through devices is a third-degree abstraction, which means that there are significant gains and losses, considering the already mentioned ideas of Flusser (2007)Flusser, Vilém (2007). O mundo codificado. São Paulo: Cosac Naify.. But the most relevant thing at this moment is how the artist says she ended up adapting to the machine’s inaccuracies, readjusting her own gestures, and, conversely, how the AI ended up generating a computational model from countless drawings by the artist, while adapting to the robot’s movements, in continuous circularity. The feedback process between them is clearly seen, and it is described in the project by the collaborative involvement of the creative action. This involvement defined as a creative catalyst causes significant changes, both sensory and cognitive, in the aesthetic experience.

FINAL CONSIDERATIONS

Needless to say, the artworks presented here do not end the proposed subject, but they suffice for our final considerations.

We have already demonstrated in the items Introduction; 1. Neuroplasticity and integration of the senses; and 2. Conformed Thought: previous studies, that:

1. In the processes of perception and representation, both the sensory and cognitive systems are used. 2. The brain adjusts to new demands, and this can happen due to its surrounding world and/or through training in learning processes. 3. Consequently, there is a feedback between internal and external systems, and this corrects inaccuracies in our senses, initially combining redundant sensory estimates and prior knowledge, but we have seen, from neuroscience and psychology, that self-configuration, adaptability and multisensory integration are also aligned. 4. As soon as we impose new conditions and experiences, the brain will conform to them. 5. It is possible to exercise cognitive skills, and improvements can already be seen from the practice of videogames, such as spatial resolution of vision, multisensory temporal processing skills, hand motor coordination, contrast sensitivity, oculomotor performance, etc. 6. And finally, since the production of images from different processes generates aesthetic, sensory and cognitive consequences, the logical processes involved in the algorithms add new questions. The techniques of classification, selection, comparison, and discrimination in the passage from an image of a thing to the model of this thing must be taken into account by aesthetic proposers. Thus, new degrees of abstraction, both of imagination and conceptualization, are required and applied in a sign feedback system. From another point of view, it is important to realize that «an episteme of causation is replaced by one of automated correlations» and «[…] which often does take the shape of a statistical hallucination» (Pasquinelli and Joler, 2020: 2Pasquinelli, Matteo and Joler, Vladan (2020). The Nooscope Manifested: Artificial Intelligence as Instrument of Knowledge Extractivism. KIM research group (Karlsruhe University of Arts and Design) and Share Lab (Novi Sad), 1 May 2020 (preprint forthcoming for AI and Society). Available in https://nooscope.ai (Accessed on January 9, 2021).).

It remains to ponder how art contributes, expands, and questions this aesthetic potential through the use of conformed thoughts. From the examples of works presented, we must consider the following.

Artists, exploring fields of possibilities of their surroundings, end up perceiving sophisticated articulations of reality that follow criteria of organization and coherence which, according to Albuquerque Vieira (2010)Albuquerque Vieira, Jorge (2010). Teoria do Conhecimento e Arte. Revista Música Hodie, 9 (2). Available in https://www.revistas.ufg.br/musica/article/view/11088/7310 (Accessed on January 9, 2021)., are associated with a root of aesthetics. In this case, since in our current times we are persistently faced with what are called Conformed Thoughts, these must be somehow altering and reconfiguring our brain plasticity. In this period of 2019-2021, we are living in a spectacularly propitious moment for new behaviors and habits, imposed by the COVID-19 pandemic. At this moment, our interactions and experiences have been restricted almost exclusively to interfaces, screens, projections, technical images, which are themselves conformed thoughts, results of concepts, models of knowledge of a culture or group; therefore, they reorganize actions determined by habits, attitudes, behaviors, and cultural practices. If, as previously mentioned, undergraduates who study extensively for three months already show signs of brain changes, there is no way to disregard that these months in which we were forced to such drastic change in habits and behaviors have altered us in any way.

Specifically, when we use image recognition systems, for example, we must remember that these are being fed by information from identifiers, classes, genders, in some cases economic status with defined socio-cultural condition, etc. The point is that neural networks cannot invent their own classes, classifiers, and constraints, and this conditions them to the recognition of the patterns in which these networks were trained. In other words, their training sets reveal the historical, geographical, racial, and socioeconomic positions of their trainers. So, as we are living in these last months according to algorithms that act in this way, we are being conformed to think and act in the way these systems were programmed. The mind itself, eager for information and curious to learn, starts acting like a learning AI network, despite the fact that the opposite is always announced, that Machine Learning networks learn from us. Breaking with this inversion is an enormous challenge. In addition to this paradox, we must realize the scope this can reach, for example, through social networks.

There are many examples of social networks that use systems that can identify people, places, objects, places, gestures, faces, sexes, economic status, and relationships. Artists are attentive and denounce these image conditioners. We must be aware that AI systems assume human visual culture and transform it into a massive and flexible training set and the greater the use of these systems, the more they will influence, condition, and shape our actions in the world. «It is time to realize that it is not the statistical model that constructs the subject, but rather the subject that structures the statistical model» (Pasquinelli and Joler, 2020: 17Pasquinelli, Matteo and Joler, Vladan (2020). The Nooscope Manifested: Artificial Intelligence as Instrument of Knowledge Extractivism. KIM research group (Karlsruhe University of Arts and Design) and Share Lab (Novi Sad), 1 May 2020 (preprint forthcoming for AI and Society). Available in https://nooscope.ai (Accessed on January 9, 2021).).

We have also presented different approaches and uses for artificial intelligence algorithms in artistic works. As we have already stated, some algorithms are able to find statistical patterns in large amounts of data and generate images that simulate representational models of different processes. From the examples shown, we can see that the nonconformed artist is not limited to applying formulas, operations, calculations, functions, and models. Their commitment is different. The artist can intercede in the process itself, subverting the very function of modeling and the algorithm, denouncing this conformed way of producing images. The artist can also train a machine and make it lose what it learned on purpose, triggering the human capacity in which remembering and forgetting are accomplices and not adversaries in the thought process. The artist can create a system that feeds back into human and non-human action, in a process capable of initiating adjustments, improvisations, mishaps, accidents, stimuli during their performances - and this strategy is an important escape valve for the pitfalls of conformed thought. The aesthetic result of all these procedures will then be relearned, reevaluated, and will reconfigure experiences through circularity and systemic feedback. And, just as the mind has human strategies and capabilities, there are non-human abilities that computational algorithms have, and both mind and algorithm bring them into this relationship for an involvement of creative action, as we intended to demonstrate.