DIRECTIONS FOR FUTURE WORK

An important area to be explored in future work is improving the stability and resilience of deep learning NLP systems in the face of noisy data of a kind that humans can filter out. Adversarial testing has been used in image recognition to address the vulnerability of DNNs to small changes in pixel input, which do not affect the classification of a figure. Jia and Liang (2017) show that small additions to suites of test sentences dramatically reduce the accuracy of a text comprehension system. We noted in Chapter 1 that Talman and Chatzikyriakidis (2019) degrade the performance of DNNs trained for natural language inference by substituting alternative lexical items and phrases for some of those in the test set.^[1] This technique allows us to identify the points at which a DNN is brittle in its classification of input. We can then work on modifying or revising its learning procedures to accommodate this sort of noise without losing precision.

It will be interesting to explore the possibility of using multi-modal training, with RL, to overcome some of the limitations in DL NLP systems. This line of research would experiment with invoking visual (or other non-linguistic) information to compensate for confusing linguistic input. Reinforcement, and other sorts of interaction could provide additional guidance in filtering this noise in the data.

Expanding training and testing to rich interactive multi-modal input will benefit DL in NLP in general. In Chapter 4, we considered recent experimental work on predicting human acceptability judgements in document contexts. A natural extension of this work would involve testing both human ratings and DNN predictions for sentences presented in non-linguistic contexts, and in dialogue settings.

In Chapter 5 we observed that applying RL in an interactive simulated visual environment permitted DNNs to achieve robust systematic generalisation of combinatorial syntactic and semantic structure. This looks like a promising avenue to explore for improving the performance of NLI systems under adversarial testing. The problems that produced poor compositional semantic generalisation in DNNs trained only on linguistic corpora may be similar, if not identical, to those that cause these systems to under perform on inference classification when trained solely on linguistic data. Humans learn to interpret new sentences, and to infer conclusions from premises, through linguistic interaction in a rich real world environment. It is reasonable to expect that a cognitively viable computational model of language learning and linguistic representation requires the same sort of input and training.

Finally, we need more intensive collaborative research with neuroscientists and psychologists to ascertain the extent to which the architecture and operation of DNNs that achieve significant success in complex NLP tasks, bear any resemblance to human learning and processing. There are two aspects to this work. The hrst is discovering whether DL can illuminate the cognitive foundations of human linguistic knowledge, beyond showing how a computational model can achieve some of the linguistic abilities that humans acquire. The second concerns the application of DL to engineering objectives. Language technology will be more effective if it performs its tasks in a way that is familiar to human users. In order to develop this sort of technology it is necessary to have deeper insight into human language learning and processing, so that we can construct DL systems that are informed by this insight.

NLP is now a flourishing area of AI, and the rise of DL methods is, in large part, responsible for the vitality and innovation that we are seeing in this work. DL has greatly expanded the horizons for cognitively interesting computational research on natural language, as well as rapid improvement in language technology across a wide range of applications. It has become a lively point of interface between computer scientists, linguists, and cognitive scientists, sharing methods and results across their disciplines. As the field continues to develop, one hopes that it will sustain a focus on foundational scientific questions. No less important is that it be guided by a firm commitment to ensuring that its technology is used for social benefit .

References

Abzianidze, L., Bjerva, J., Evang, K., Haagsma, H.. van Noord, R., Lud- mann, P., ... Bos, J. (2017). The parallel meaning bank: Towards a multilingual corpus of translations annotated with compositional meaning representations. ArXiv preprint arXiv:1702.03964 ■

Adger, D. (2003). Core syntax: A minimalist approach. United Kingdom: Oxford University Press.

Bahdanau, D., Cho, К., X Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. ArXiv, abs/1409.0473.

Baroni, M., Bernardini, S., Ferraresi, A., X Zanchetta, E. (2009). The wacky wide web: A collection of very large linguistically processed web-crawled corpora. Language Resources and Evaluation, ^3(3), 209-226.

Berger, J. O. (1985). Statistical decision theory and Bayesian analysis (2nd ed.). New York: Springer-Verlag.

Bernardy, J.-P., Blanck, R., Chatzikyriakidis, S., Lappin, S., X Maskha- rashvili, A. (2019a). Bayesian inference semantics: A modelling system and a test suite. In Proceedings of the eighth joint conference on lexical and computational semantics (*SEM 2019) (pp. 263-272). Minneapolis, Minnesota: Association for Computational Linguistics.

Bernardy, J.-P., Blanck, R., Chatzikyriakidis, S., Lappin, S., X Maskha- rashvili, A. (2019b). Predicates as boxes in Bayesian semantics for natural language. In Proceedings of the 22nd Nordic conference on computational linguistics (pp. 333-337). Turku, Finland: Linkoping University Electronic Press.

Bernardy, J.-P., X Lappin, S. (2017). Using deep neural networks to learn syntactic agreement. Linguistic Issues in Language Technology, 15, 1-15.

Bernardy, J.-P., Lappin, S., X Lau, J. H. (2018). The influence of context on sentence acceptability judgements. In Proceedings of the 56th annual meeting of the association for computational linguistics (ACL 2018) (pp. 456-461). Melbourne, Australia.

Bizzoni, Y., X Lappin, S. (2017). Deep learning of binary and gradient judgements for semantic paraphrase. In IWCS 2017 - 12th international conference on computational semantics - Short papers.

Bizzoni, Y., X Lappin, S. (2019). The effect of context on metaphor paraphrase aptness judgments. In Proceedings of the 13th international conference on computational semantics - Long papers (pp. 165 175). Gothenburg, Sweden.

Blank, I., Balewski. Z., Mahowald, K.. Sz Fedorenko, E. (2016). Syntactic processing is distributed across the language system. Neuroimage, 127, 307-323.

Bowman, S. R., Angeli, G., Potts, C., &: Manning, C. D. (2015, September). A large annotated corpus for learning natural language inference. In Proceedings of the 2015 conference on empirical methods in natural language processing (pp. 632-642). Lisbon, Portugal: Association for Computational Linguistics.

Bowman, S. R., Gauthier, J., Rastogi, A., Gupta, R., Manning, C. D., $z Potts, C. (2016, August). A fast unified model for parsing and sentence understanding. In Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 1466-1477). Berlin, Germany: Association for Computational Linguistics.

Bresnan, J. (2001). Lexical-functional syntax. Malden, MA: Blackwell.

Bringsjord, S., &: Govindarajulu, N. S. (2020). Artificial intelligence. In

E. N. Zalta (Ed.), The Stanford encyclopedia of philosophy (Summer 2020 ed.). Metaphysics Research Lab, Stanford University.

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.. Dhariwal, P.,

... Anrodei, D. (2020). Language models are few-shot learners. ArXiv, abs/2005.14165.

Causse, M., Peysakhovich, V., &; Fabre, E. F. (2016). High working memory load impairs language processing during a simulated piloting task: An ERP and pupillometry study. Frontiers in Human Neuroscience, 10, 240.

Chen, D.. &: Manning, C. (2014). A fast and accurate dependency parser using neural networks. In Proceedings of the 20Ц conference on empirical methods in natural language processing (EMNLP) (pp. 740-750).

Chen, S., Yao, T., Sz Jiang, Y.-G. (2019). Deep learning for video captioning: A review. In Proceedings of the twenty-eighth international joint conference on artificial intelligence, IJCAI-19 (pp. 6283-6290). International Joint Conferences on Artificial Intelligence Organization.

Cho, K., van МегпёпЬоег, В.. Gulcehre, C., Bahdanau, D., Bougares,

F. , Schwenk, H., Sz Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1724-1734). Doha, Qatar: Association for Computational Linguistics.

Choi, J., Yoo, К. M., &; Lee, S.-g. (2018). Learning to compose task- specific tree structures. In AAAI conference on artificial intelligence.

Chomsky, N. (1957). Syntactic structures. The Hague: Mouton.

Chomsky. N. (1965). Aspects of the theory of syntax. Cambridge, MA: The MIT Press.

Chomsky. N. (1981). Lectures on government and binding. Dordrecht, Holland: Foris Publications.

Chomsky. N. (1995). The minimalist program. Cambridge, MA: The MIT Press.

Clark, A. (2003). Combining distributional and morphological information for part of speech induction. In Proceedings of the tenth conference on European chapter of the association for computational linguistics (Vol. 1, pp. 59-66). USA: Association for Computational Linguistics.

Clark, A. (2015). Canonical context-free grammars and strong learning: Two approaches. In Proceedings of the 14th meeting on the mathematics of language (MoL 2015) (pp. 99-111).

Clark, A., Fox, C., & Lappin, S. (Eds.). (2010). The handbook of computational linguists and natural language processing. Malden, MA and Oxford: Wiley-Blackwell.

Clark, A., & Lappin, S. (2010). Unsupervised learning and grammar induction. In A. Clark, C. Fox, &: S. Lappin (Eds.), The handbook of computational linguists and natural language processing (pp. 197— 220). Malden, MA and Oxford: Wiley-Blackwell.

Clark, A., &; Lappin, S. (2011). Linguistic nativism and the poverty of the stimulus. Malden, MA and Oxford: Wiley-Blackwell.

Clark, A., Lappin, S. (2013). Complexity in language acquisition. Topics in Cognitive Science, 5(1), 89-110.

Clark, A., & Yoshinaka, R. (2014). Distributional learning of parallel multiple context-free grammars. Machine Learning, 55(1-2), 5-31.

Clark, S., Curran, J. R. (2007). Wide-coverage efficient, statistical parsing with CCG and log-linear models. Computational Linguistics, 33(4), 493-552.

Coecke, B., Sadrzadeh, M., Clark, S. (2010). Mathematical foundations for a compositional distributional model of meaning. Lambek Festschrift, Linguistic Analysis, 36.

Collins. C., &z Stabler, E. (201G). A formalization of minimalist syntax. Syntax, 19.

Cooper, R., Dobnik. S., Larsson, S., $z Lappin, S. (2015). Probabilistic type theory and natural language semantics. Linguistic Issues in Language Technology, 10.

Cooper, R., $z Ginzburg, J. (2015). Type theory with records for natural language semantics. In S. Lappin &: C. Fox (Eds.), The handbook of contemporary semantic theory (pp. 375-407). John Wiley &z Sons, Ltd.

Crain, S., Iz Thornton, R. (1998). Investigations in universal grammar: A guide to experiments on the acquisition of syntax and semantics. MIT Press.

Davidson, D. (19G7). Truth and meaning. Synthese, 17(1), 304-323.

de Marneffe, М.-C., MacCartney, B., Sz Manning, C. D. (200G, May). Generating typed dependency parses from phrase structure parses. In Proceedings of the fifth international conference on language resources and evaluation (LREC 06). Genoa, Italy: European Language Resources Association (ELRA).

Devlin, J., Chang, M.-W., Lee, K., Sz Toutanova, K. (2019, June). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers) (pp. 4171-418G). Minneapolis, Minnesota: Association for Computational Linguistics.

Dubhashi, D., Iz Lappin, S. (2017). AI dangers: Imagined and real. Communications of the ACM, 60(2), 43-45.

Dyer, C., Kuncoro, A.. Ballesteros, M., Iz Smith, N. A. (201G, June). Recurrent neural network grammars. In Proceedings of the 2016 conference of the north American chapter of the association for computational linguistics: Human language technologies (pp. 199— 209). San Diego, CA: Association for Computational Linguistics.

Ek, A., Bernardy, J.-P., $z Lappin, S. (2019). Language modeling with syntactic and semantic representation for sentence acceptability predictions. In Proceedings of the 22nd Nordic conference on computational linguistics (pp. 76-85). Turku, Finland.

Elman, J. L. (1990). Finding structure in time. Cognitive Science, Ц (2), 179-211.

Fodor, J. A. (2000). The mind doesn’t work that way: The scope and limits of computational psychology. MIT Press.

Fodor, J. A., X Pylyshyn, Z. W. (1988). Connectionism and cognitive architecture: A critical analysis. In S. Pinker X J. Mehler (Eds.), Connections and symbols (pp. 3-71). Cambridge, MA, USA: MIT Press.

Fox, C., X Lappin, S. (2010). Expressiveness and complexity in under- specified semantics. Lambek Festschrift, Linguistic Analysis, 36, 385-417.

Francois-Lavet, V., Henderson, P., Islam, R., Bellemare, M. G., X Pineau, J. (2018). An introduction to deep reinforcement learning. Foundations and Trends in Machine Learning, 11(3-4), 219-354.

Francez, N., X Dyckhoff, R. (2010). Proof-theoretic semantics for a natural language fragment. Linguistics and Philosophy, 33(6), 447- 477.

Gibson, E., X Fedorenko, E. (2013). The need for quantitative methods in syntax and semantics research. Language and Cognitive Processes, 28(1-2), 88-124.

Gibson, E., Piantadosi, S. T., X Fedorenko, E. (2013). Quantitative methods in syntax/semantics research: A response to Sprouse and Almeida (2013). Language and Cognitive Processes, 28(6), 229- 240.

Gold, E. M. (19G7). Language identification in the limit. Information and Control, 10(6), 447-474.

Goldberg, Y. (2019). Assessing BERT’s syntactic abilities. ArXiv, abs/1901.05287.

Gomez-Rodriguez, С., X Vilares, D. (2018). Constituent parsing as sequence labeling. ArXiv. (arXiv: 1810.08994)

Goodfellow, I., Bengio, Y., X Courville, A. (2016). Deep learning. The MIT Press.

Goodman, J. (2001, 10). A bit of progress in language modeling. Computer Speech & Language, 15, 403-434.

Grefenstette, E., Sadrzadeh, M., Clark, S., Coecke, В., X Pulman, S. (2011). Concrete sentence spaces for compositional distributional models of meaning. In Proceedings of the ninth international conference on computational semantics (IWCS 2011).

Gulordava, K., Bojanowski, P., Grave, E., Linzen, T., X Baroni, M. (2018). Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long papers) (pp. 1195-1205). New Orleans, LA: Association for Computational Linguistics.

He, S., Liao, W., Tavakoli, H.. Yang, M., Rosenhahn, B., X Pugeault, N. (2020). Image captioning through image transformer. ArXiv, 1-17.

Herdade, S., Kappeler, A., Boakye, K., X Soares, J. (2019). Image captioning: Transforming objects into words. In Neural information processing systems.

Hewitt, J., X Manning, C. D. (2019, June). A structural probe for finding syntax in word representations. In Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers) (pp. 4129-4138). Minneapolis, MN: Association for Computational Linguistics.

Hill, F., Lampinen, A. K., Schneider, R., Clark, S., Botvinick, M., McClelland, J. L., X Santoro, A. (2020). drivers of systematicity and generalization in a situated agent. In 8th international conference on learning representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.

Hill, F., Reichart, R., X Korhonen, A. (2015). SimLex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, fl, 665-695.

Hill, F., Tieleman, O., von Glehn, T., Wong, N., Merzic, H., X Clark, S. (2020). Grounded language learning fast and slow. ArXiv.

Hochreiter, S., X Schmidhuber, J. (1997, 12). Long short-term memory. Neural Computation, 9, 1735-1780. doi: 10.1162/neco.l997.9.8. 1735

Hu, J., Gauthier, J., Qian, P., Wilcox, E., X Levy, R. (2020, July). A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th annual meeting of the association for computational linguistics (pp. 1725-1744). Online: Association for Computational Linguistics.

Ito, A., Corley, M., X Pickering, M. J. (2018). A cognitive load delays predictive eye movements similarly during 11 and 12 comprehension. Bilingualism: Language and Cognition, 21(2), 251-264.

Jia, R., X Liang, P. (2017). Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 conference on empirical methods in natural language processing (pp. 2021— 2031). Copenhagen, Denmark: Association for Computational Linguistics.

Johnson, D., X Lappin, S. (1999). Local constraints vs. economy. Stanford, CA: CSLI Publications.

Johnson, D., Postal, P. (1981). Arc pair grammar. Princeton, NJ: Princeton University Press.

Joshi, A. K., Schabes, Y. (1997). Tree-adjoining grammars. In G. Rozenberg A. Salomaa (Eds.), Handbook of formal languages (Vol. 3, pp. 69-124). Berlin, New York: Springer.

Joshi, A. K., Shanker, К. V., &; Weir, D. (1990). The convergence of mildly context-sensitive grammar formalisms (Tech. Rep.). Philadelphia, PA: Department of Computer and Information Science, University of Pennsylvania.

Jozefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., Wu, Y. (2016). Exploring the limits of language modeling. ArXiv, abs/1602.02410.

Kamp, H., &: Reyle, U. (1993). From discourse to logic: Introduction to model-theoretic semantics of natural language, formal logic and discourse representation theory (Vol. 42). Dordrecht: Springer.

Kaplan, R. M., V Bresnan, J. (1982). Lexical-functional grammar: A formal system for grammatical representation. In J. Bresnan (Ed.), The mental representation of grammatical relations (pp. 173-281). Cambridge, MA: MIT Press.

Karpathy, A., h Fei-Fei, L. (2015). Deep visual-semantic alignments for generating image descriptions. ArXiv.

Kempson, R., Meyer-Viol, W., &: Gabbay, D. (2000). Dynamic syntax: The flow of language understanding. Wiley-Blackwell.

Kim. Y. (2014). Convolutional neural networks for sentence classification. In Proceedings of the 201 f conference on empirical methods in natural language processing (EMNLP) (pp. 1746-1751). Doha, Qatar: Association for Computational Linguistics.

Klein. D., Manning, C. D. (2003a). Accurate unlexicalized parsing. In Proceedings of the fist annual meeting of the association for computational linguistics (pp. 423-430). Sapporo, Japan: Association for Computational Linguistics.

Klein. D., V Manning, C. D. (2003b). Fast exact inference with a factored model for natural language parsing. In S. Becker, S. Thrun, V K. Obermayer (Eds.), Advances in neural information processing systems 15 (pp. 3-10). MIT Press.

Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., ... Herbst, E. (2007). Moses: Open source toolkit for statistical machine translation. In Proceedings of the foth annual meeting of the association for computational linguistics companion volume proceedings of the demo and poster sessions (pp. 177-180). Prague, Czech Republic.

Kuncoro, A., Dyer, C.. Hale, J., Yogatama, D., Clark, S., &; Blunsom. P. (2018). LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better. In Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 1426-1430). Melbourne, Australia: Association for Computational Linguistics.

Kuncoro, A., Dyer, C., Rimell, L., Clark, S., Blunsom. P. (2019, July). Scalable syntax-aware language models using knowledge distillation. In Proceedings of the 51th annual meeting of the association for computational linguistics (pp. 3472-3484). Florence, Italy: Association for Computational Linguistics.

Kuncoro, A., Kong, L., Fried, D., Yogatama, D., Rimell, L., Dyer, C., Y Blunsom, P. (2020). Syntactic structure distillation pretraining for bidirectional encoders. ArXiv.

Lake, B., &: Baroni, M. (2018). Generalization without systematic- ity: On the compositional skills of sequence-to-sequence recurrent networks. In J. Dy A. Krause (Eds.), Proceedings of machine learning research (Vol. 80, pp. 2873-2882). Stockholm Sweden: PMLR.

Lambek, J. (1958). The mathematics of sentence structure. American Mathematical Monthly.

Lambek, J. (2008). Pregroup grammars and Chomsky’s earliest examples. Journal of Logic, Language and Information, 17, 141- 160.

Lappin, S. (2015). Curry typing, polymorphism, and fine-grained inten- sionality. In S. Lappin C. Fox (Eds.), The handbook of contemporary semantic theory (pp. 408-428). John Wiley Sons, Ltd.

Lappin, S., Y Lau, J. H. (2018). Gradient probabilistic models vs categorical grammars: A reply to Sprouse et al. (2018). In The science of language (Ted Gibson’s Blog).

Lappin, S., Shieber, S. (2007). Machine learning theory and practice as a source of insight into universal grammar. Journal of Linguistics, 43, 393-427.

Lau, J. H., Armendariz, C., Lappin, S., Purver, M., !sz Shu, C. (2020). How furiously can colorless green ideas sleep? Sentence acceptability in context. Transactions of the Association for Computational Linguistics, 8, 296-310. doi: 10.1162/tacl_a_00315

Lau, J. H.. Baldwin, T., &: Cohn, T. (2017). Topically driven neural language model. In Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 355-365). Vancouver, Canada.

Lau, J. H., Clark, A., !k Lappin, S. (2014). Measuring gradience in speakers’ grammaticality judgements. In Proceedings of the 36th annual conference of the cognitive science society (pp. 821-826). Quebec City, Canada.

Lau, J. H., Clark, A., V Lappin, S. (2015). Unsupervised prediction of acceptability judgements. In Proceedings of the joint conference of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing of the Asian Federation of Natural Language Processing (ACL-IJCNLP 2015) (pp. 1618-1628). Beijing, China.

Lau, J. H., Clark, A., V Lappin, S. (2017). Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, jl. 1202-1241.

Lecun, Y., Kavukcuoglu, K., &: Farabet, C. (2010, 05). Convolutional networks and applications in vision. In ISC AS 2010 - 2010 IEEE international symposium on circuits and systems: Nano-bio circuit fabrics and systems (pp. 253-256).

Lewis, M., V Steedman, M. (2013). Combined distributional and logical semantics. Transactions of the Association for Computational Linguistics, 1, 179-192.

Linzen, T., Chrupala, G., V Alishahi, A. (Eds.). (2018). Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP. Brussels, Belgium: Association for Computational Linguistics.

Linzen, T., Chrupala, G., Belinkov, Y., V Hupkes, D. (Eds.). (2019, August). Proceedings of the 2019 ACL workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP. Florence, Italy: Association for Computational Linguistics.

Linzen, T., Dupoux, E., V Goldberg, Y. (2016). Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4, 521-535.

Lu, J., Goswami, V., Rohrbach, M., Parikh, D., V Lee, S. (2020). 12-in- 1: Multi-task vision and language representation learning. In 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), 10434-10443.

Maillard, J., Clark, S., &; Yogatama, D. (2019). Jointly learning sentence embeddings and syntax with unsupervised Tree-LSTMs. Natural Language Engineering, 25(4), 433-449.

Marcus, G. (2001). The algebraic mind: Integrating connectionism and cognitive science. Cambridge, MA: MIT Press.

Marcus, M. P., Santorini, B., &: Marcinkiewicz, M. A. (1993). Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2), 313-330.

Marelli, M., Bentivogli, L.. Baroni, M., Bernardi, R., Menini, S., Zam- parelli, R. (2014). SenrEval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014) (pp. 1-8). Dublin, Ireland: Association for Computational Linguistics.

Marvin, R., Linzen, T. (2018). Targeted syntactic evaluation of language models. In Proceedings of the 2018 conference on empirical methods in natural language processing (pp. 1192-1202). Brussels, Belgium: Association for Computational Linguistics.

McCoy, R. T., Frank, R.. Sz Linzen, T. (2020). Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence- to-sequence networks. Transactions of the Association for Computational Linguistics, 8, 125-140.

Mikolov, T. (2012). Statistical language models based on neural networks (Unpublished doctoral dissertation). Brno University of Technology.

Mikolov, T., Kombrink, S., Deoras, A., Burget, L., &z Eernocky, J. (2011). RNNLM - Recurrent neural network language modeling toolkit. In IEEE automatic speech recognition and understanding workshop. Hawaii, US.

Mikolov, T., Sutskever, I., Chen, K., Corrado, G., &; Dean, J. (2013). Distributed representations of words and phrases and their conrpo- sitionality. In Proceedings of the 26th international conference on neural information processing systems - volume 2 (pp. 3111-3119). Red Hook, NY, USA: Curran Associates Inc.

Mitchell, J., Sz Lapata, M. (2008). Vector-based models of semantic composition. In Proceedings of ACL-08: HLT (pp. 236-244). Columbus, Ohio: Association for Computational Linguistics.

Montague, R. (1974). Formal philosophy: Selected papers of Richard Montague. New Haven, CT/London, UK: Yale University Press. (Edited with an introduction by R. H. Thomason)

Moortgat, M. (1997). Categorial type logics. In J. van Benthem A. ter Meulen (Eds.), Handbook of logic and language (pp. 93-177). North Holland / Elsevier.

Morrill, G. (1994). Type logical grammar: Categorial logic of signs. Springer.

Nivre, J., Agic, Z., Ahrenberg, L., et al. (2017). Universal dependencies 2.0. (LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University)

Nivre, J., De Marneffe, М.-C., Ginter, F., Goldberg, Y., Hajic, J., Manning, C. D., ... others (2016). Universal dependencies vl: A multilingual treebank collection. In LREC.

Nivre, J., Hall, J., Nilsson, J., Chanev, A., Ervigit, G., Kiibler, S., ... Marsi, E. (2007). Maltparser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13, 95-135.

Pado, S., &: Lapata, M. (2007). Dependency-based construction of semantic space models. Computational Linguistics, 33(2), 161-199.

Park, H., Kang, J.-S., Choi, S., Lee, M. (2013). Analysis of cognitive load for language processing based on brain activities. In M. Lee, A. Hirose, Z.-G. Hou, &: R. M. Kil (Eds.), Neural information processing (pp. 561-568). Berlin, Heidelberg: Springer Berlin Heidelberg.

Pauls, A., &: Klein, D. (2012). Large-scale syntactic language modeling with treelets. In Proceedings of the 50th annual meeting of the association for computational linguistics (pp. 959-968). Jeju, Korea.

Pennington, J., Soeher, R., Sz Manning, C. (2014). GloVe: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543). Doha, Qatar: Association for Computational Linguistics.

Pereira, F. (2000). Formal grammar and information theory: Together again? In Philosophical transactions of the royal society (p. 1239- 1253). London: Royal Society.

Pollard, C., &; Sag, I. A. (1994). Head-driven phrase structure grammar. Stanford, CA: CSLI Publications.

Pullum, G. K., & Scholz, В. C. (2001). On the distinction between model-theoretic and generative-enumerative syntactic frameworks. Logical aspects of computational linguistics: fth international conference, Le Croisic, France, June 27-29, 2001, Proceedings.

Radford, A., Narasimhan. K., Salimans, T., &: Sutskever, I. (2018). Improving language understanding by generative pre-training. Ope- nAI.

Richardson. К., Hu, H., Moss, L. S., Sz Sabharwal, A. (2020). Probing natural language inference models through semantic fragments. In The thirty-fourth AAAI conference on artificial intelligence, AAAI 2020, the thirty-second innovative applications of artificial intelligence conference, IAAI 2020, the tenth AAAI symposium on educational advances in artificial intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 (pp. 8713-8721). AAAI Press.

Rumelhart, D. E., McClelland, J. L., &; PDP Research Group, C. (Eds.). (1986). Parallel distributed processing: Explorations in the microstructure of cognition, Vol. 1: Foundations. Cambridge, MA, USA: MIT Press.

Sehrimpf, M., Blank, I., Tuckute, G., Kauf, C., Hosseini, E. A., Kan- wisher, N., ... Fedorenko, E. (2020). Artificial neural networks accurately predict language processing in the brain. bioRxiv.

Shieber, S. M. (1985). Evidence against the context-freeness of natural language. Linguistics and Philosophy, 5(3), 333-343.

Shieber, S. M. (2004). The turing test: Verbal behavior as the hallmark of intelligence. The MIT Press.

Shieber, S. M. (2007). The turing test as interactive proof. A^Tous, fl (4), G8G-713.

Shieber, S. M., h Schabes, Y. (1990). Synchronous tree-adjoining grammars. In 13th international conference on computational linguistics, COLING 1990, University of Helsinki, Finland, August 20-25, 1990 (pp. 253-258).

Smolensky, P. (1990). Tensor product variable binding and the representation of symbolic structures in connectionist. systems. Artificial Intelligence, 46(1-2), 159-21G.

Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., &; Ng, A. Y. (2014). Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics, 2.

Socher, R., Pennington, J., Huang, E. H., Ng, A. Y., Manning, C. D. (2011, July). Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the 2011 conference on empirical methods in natural language processing (pp. 151-161). Edinburgh, Scotland, UK: Association for Computational Linguistics.

Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., &; Potts, C. (2013, October). Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing (pp. 1631-1642). Seattle, WA, USA: Association for Computational Linguistics.

Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert-Voss, A., Wu, J., ... Wang, J. (2019). Release strategies and the social impacts of language models. ArXiv, abs/1008.09203.

Sprouse, J., V Almeida, D. (2013). The empirical status of data in syntax: A reply to Gibson and Fedorenko. Language and Cognitive Processes, 28, 222-228.

Sprouse, J., Schritze, С., V Almeida, D. (2013). A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001-2010. Lingua, 134, 219-248.

Sprouse, J., Yankama, B., Indurkhya, S., Fong, S., h Berwick, R. C. (2018). Colorless green ideas do sleep furiously: Gradient acceptability and the nature of the grammar. The Linguistic Review, 35(3), 575-599.

Steedman, M. (2000). The syntactic process. Cambridge, MA: MIT Press.

Sutton, P. (2017, 06). Probabilistic approaches to vagueness and semantic competency. Erkenntnis, 83, 1-30.

Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257-285.

Tai, K. S., Socher, R., V Manning, C. D. (2015, July). Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: Long papers) (pp. 1556-1566). Beijing, China: Association for Computational Linguistics.

Talman, A., V Chatzikyriakidis, S. (2019, August). Testing the generalization power of neural network models across NLI benchmarks.

In Proceedings of the 2019 ACL workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP (pp. 85-94). Florence, Italy: Association for Computational Linguistics.

Turing, A. M. (1950). Computing machinery and intelligence. Mind. 59(October), 433-460.

Turney, P., &; Pantel, P. (2010). From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 57.

Valiant, L. G. (1984). A theory of the learnable. In Proceedings of the sixteenth annual ACM symposium on theory of computing (STOC ’84) (pp. 436-445). New York, NY, USA: ACM Press, doi': http://doi.acm.org/10.1145/800057.808710

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... Polosukhin, I. (2017). Attention is all you need. In Proceedings of the 31st international conference on neural information processing systems (pp. 6000-6010). Red Hook, NY, USA: Curran Associates Inc.

Vinyals, O., Toshev, A., Bengio, S., V Erhan, D. (2014). Show and tell: A neural image caption generator. ArXiv, abs/1411-4555.

Wang, A., Singh, A., Michael, J., Hill, F., Levy, О., V Bowman, S. (2018). GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP (pp. 353-355). Brussels, Belgium: Association for Computational Linguistics.

Warstadt, A., Singh, A., Bowman, S. (2019). Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7(0).

Williams, A., Drozdov, A., V Bowman, S. R. (2018). Do latent tree learning models identify meaningful structure in sentences? Transactions of the Association for Computational Linguistics, 6, 253- 267.

Williams, A., Nangia, N., V Bowman, S. (2018). A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 conference of the north American chapter of the association for computational linguistics: Human language technologies, Volume 1 (long papers) (pp. 1112-1122). New Orleans, LA: Association for Computational Linguistics.

Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., ... Bengio, Y. (2015). Show, attend and tell: Neural image caption generation with visual attention. In F. Bach &; D. Blei (Eds.), Proceedings of machine learning research (Vol. 37, pp. 2048-2057). Lille, France: PMLR.

Yang, Z., Dai, Z., Yang, Y., Carbonell, J. G., Salakhutdinov, R.. &: Le, Q. V. (2019). XLNet: Generalized autoregressive pretraining for language understanding. ArXiv, abs/1906.08237.

Yogatama, D., Blunsom, P., Dyer, C., Grefenstette, E., Y Ling, W. (2017). Learning to compose words into sentences with reinforcement learning. In 5th international conference on learning representations, ICLR 2017, Toulon, France, April 24 -26, 2017, Conference track proceedings.

Zhang, W. E., Sheng, Q. Z., Alhazmi, A., & Li, C. (2020). Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology, 11(3).

• [1] See Zhang, Sheng, Alhazmi, and Li (2020) for an extensive overview of adversarial testing methods in deep learning NLP.