The identification and segmentation of buildings in remote sensing imagery has consistently been a important point of academic research. This work highlights the effectiveness of using diverse datasets and advanced representation learning models for the purpose of building segmentation in remote sensing images. By fusing various datasets, we have broadened the scope of our learning resources and achieved exemplary performance across several datasets. Our innovative joint training process demonstrates the value of our methodology in various critical areas such as urban planning, disaster management, and environmental monitoring. Our approach, which involves combining dataset fusion techniques and prompts from pre-trained models, sets a new precedent for building segmentation tasks. The results of this study provide a foundation for future exploration and indicate promising potential for novel applications in building segmentation field.