Maize (Zea mays L.) constitutes 36\% (782 metric tonnes) of the global grain production and is one of the most versatile crops that grows under varied climatic conditions, making it a staple food in most countries. It contributes nearly 9\% to the Indian food basket and more than 100 billion INR to the agricultural GDP. Due to climate change and growing demand, food safety and security are greatly affected. We endeavour to develop a deep learning-based technique that can aid farmers in improving yield by identifying stress (drought and nitrogen-based). Concretely, we aim to develop an end-to-end pipeline in conjunction with farmers and agricultural researchers to, (1) perform data collection of RGB and multispectral data using drones at various stages of growth and stress (2) propose deep learning-based methods that can identify various kinds of stress and recommend required action and, (3) work with the farmers and agricultural researchers to deploy this technology to aid their production.