Seminar Details

Deep language models are ground-breaking techniques amazingly effective in many tasks, while at the same time
they are also fragile and constrain on sequence to sequence learning tasks. This talk presents my recent research on
“how to move forward with BERT and GPT-2” by introducing structures to deep neural networks. The first is to
upgrade Transformer’s attention connections using the existing intrinsic structures in data. This leads to
Transformer-XH (eXtra-Hop), which upgrades Transformer with inter-document (hop) attention connections to share
information across multiple pieces of evidence. The second is to bring in external structured semantics from
knowledge graphs to neural language modeling and generation. We ground the texts to an existing knowledge
graph, form a latent semantic space using the grounded semantics, and aid the language modeling with a simulated
ConceptFlow in the latent semantic space. These techniques achieved significant better performances than BERT
and GPT-2 on multi-hop question answering and conversation response generation, respectively.