Answer Set Programming (ASP) and Large Language Models (LLMs) have emerged as powerful tools in Artificial Intelligence, each offering unique capabilities in knowledge representation and natural language processing, respectively. In this paper, we combine the strengths of the two paradigms with the aim of improving the structured representation of complex knowledge encoded in natural language. In a nutshell, the structured representation is obtained by combining syntactic structures extracted by LLMs and semantic aspects encoded in the knowledge base. The interaction between ASP and LLMs is driven by a YAML file specifying prompt templates and domain-specific background knowledge. The proposed approach is evaluated using a set of benchmarks based on a dataset obtained from problems of ASP Competitions. The results of our experiment show that ASP can sensibly improve the F1-score, especially when relatively small models are used.